Predicting Stance Change Using Modular Architectures (2020)

Abstract: The ability to change a person’s mind on a given issue depends both on the arguments they are presented with and on their underlying perspectives and biases on that issue. Predicting stance changes require characterizing both aspects and the interaction between them, especially in realistic settings in which stance changes are very rare. In this paper, we suggest a modular learning approach, which decomposes the task into multiple modules, focusing on different aspects of the interaction between users, their beliefs, and the arguments they are exposed to. Our experiments show that our modular approach archives significantly better results compared to the end-to-end approach using BERT over the same inputs.

## introduction one of the main drivers of social interaction is convincing people to adopt new ideas and perspectives. understanding how to make compelling arguments that would achieve this goal has been studied extensively in psychology and the social sciences @xcite , and more recently by the nlp community. most of these works focus on analyzing argumentative text in isolation @xcite @xcite . while the quality of arguments can potentially be judged in isolation based on their merits, their effectiveness, when it comes to convincing readers, has to be considered in conjunction with the subjective perspectives and biases of these users. for example, social psychology studies have shown that ideological stances are highly correlated with different moral arguments preferences @xcite @xcite , while @xcite studied the relation between personality type and factual vs. emotional arguments. @xcite studied linguistic properties of convincing arguments, conditioned on users' political and religious convictions. this work aims to study the effectiveness of debate arguments, with respect to the biases of users reading them. our goal is to mimic a realistic setting, in which only partial information about these users and their behaviors is available. to accommodate this setting, we use data from the website debate.org, used in several previous studies @xcite @xcite , and formulate a new classification task, changemystance, (cms) 1 , predicting whether a specific user would change their stance after being exposed to two conflicting texts: one that does not support the voting user's views and another one that does. as people do not easily change their minds when exposed to other perspectives, this formulation results in a highly challenging task with only about 6% of users persuaded by the arguments. this formulation is different than the one proposed by @xcite , which also studied users' bias, as explained in section 4. our main intuition is that successfully approaching the cms task hinges on modeling the interactions between the arguments made by each side and the biases of the user exposed to them. for example, when debating self-driving cars, a computer scientist might focus on technical challenges while an economist would be more attuned to arguments discussing the implications on the job market. similar to durmus and cardie (2018), we rely on users' self-reported information, which is often incomplete as users do not report their views in a consistent way and omit details. instead, we follow the observation that users' bias can also be reflected by their behavior, and take a representation-learning approach, mapping the observed users' information to a latent space based on their agreement patterns. we exploit the fact that the cms task naturally decomposes into sub-tasks, each capturing a different aspect of the problem. instead of an end-to-end formulation, we use modular learning, in which the different aspects are captured by different modules, combined later to support the final cms decision. we take a hierarchical task decomposition approach, starting with elementary modules that shape users' representations based on their agreement patterns (agreement task). we characterize the interaction between users and textual content by defining three auxiliary tasks: characterizing the users' initial bias on the issue, their final stance, and argument preferences (before, after, and arguments, resp.). we consider several ways of using these modules to support the cms task. the first is the traditional multi-task learning approach @xcite in which the input representation is shaped by both the auxiliary and final tasks. however, the dependency between the cms task and its modules can also build on the output of the modules. for example, cms can be approximated by combining the modules for predicting the user's initial bias on the topic and their argument preference (i.e., before + arguments). if the user finds the arguments of the side that conflicts with their original perspective more appealing, they are more likely to be persuaded. we design several hierarchical modular networks, and compare them to a strong end-to-end baseline based on bert @xcite . our results demonstrate the importance of characterizing the user bias using the modules leading to significant improvements. ## related work understanding the properties of persuasive arguments and identifying them in text have been widely studied in the nlp community @xcite @xcite @xcite @xcite @xcite @xcite . most relevant to our work is the changemyview (cmv) task, introduced by @xcite , predicting stance changes in reddit forums. similar to our work, @xcite highlight the importance of modeling the user's belief when analyzing persuasive language. in their work, @xcite approach the cmv problem using features that characterize the users' psychological attributes. we suggest a modular approach, adapting the modular architecture suggested by @xcite for sequence tagging to detecting user-specific persuasive text. closest to our settings is @xcite , which follows the cmv line of research and proposed an end to end neural network with a modular attention mechanism. it consists on two modules, one that models the malleable parts of the original poster's arguments, and another that focuses the difference in content between both authors. ## the dataset and preprocessing the changemystance task is defined over two texts with opposing perspectives on an issue, and a user indicating if their stance on the issue changed after reading the arguments in the texts. the debates, collected from www.debate.org , are between two contenders: the initiator (in) proposing the debate question and the challenger (ch), offering the opposing view. debates consist of a series of rounds, in which contenders respond to the arguments made by the other contender in the previous round. at the end of the debate, users can express their preferences by voting, (as depicted in fig. 1 ) by casting a 0 (vote for in), 2 (voter for ch) or 1 (tied) in the following categories: u before (voter's preference before the debate), u after (voter's preference after the debate.), u arguments (which side made better arguments?), u conduct (which side had better conduct?), u writing (which side had better writing skills?). users can share information on their profile page, including attributes such as age, political ideology and education. they can also provide their stances over 48 "big"-issues (e.g., abortion, healthcare, etc.), and provide an open-ended textual summary. we refer to the first two (big issues stances and profile attributes) as uprofile and use usummary to refer to the textual summaries, represented by applying a pretrained bert encoder on that text. preprocessing we filter debates containing arguments by only one side, or low engagement (less than 3 votes). after filtering we are left with 12901 debates and 4380 different voters participating in them. we represent the debate text by fine-tuning the bert @xcite model on our data, treating each round as a sentence capped to a size of 510 tokens. in order to represent the text of the posts we use a fixed bert encoding of the first 510 tokens of the debate (same for usummary). ## problem definition and decomposition in this section we will define the changemystance task, its supporting sub-problems, and explain how it differs from similar persuasion-detection tasks. changemystance is a binary classification problem, each instance is a 4-@xmath0, t 1 , t 2 , y >, where u is a user representation, t 1 and t 2 are two pieces of text with opposing views. as a convention, we assume that t 2 is the text expressing u's initial stance on the topic. the label @xmath1, 1} captures the stance change, i.e., @xmath2, i.e., when u @xmath3. when the user is initially undecided, i.e., u @xmath4, we allow both sides to change the user stance. changes to an undecided view are also considered as stance changes. comparison with @xcite the interaction between users' bias and the type of arguments they find convincing over the debate.org dataset was previously studied @xcite . they defined controlled studies in which the users were known to change their mind, or required models for predicting the stance preference rather than stance change. however, our task formulation is very different. we focus on modeling the realistic setting of persuasion detection, in which only a small fraction of users will be convinced to change their mind. in our view, this is a more natural setting. for example, when launching a campaign on social media promoting better health choices, policy makers have to consider which arguments will be convincing based on limited information. we specifically focused on these challenging settings, as a way to evaluate different strategies to represent user's beliefs and its interaction with argumentative content. ## models in the previous section we introduced the cms task, and four related sub-tasks, designed to support it. we use the term module to refer to the neural architecture trained for each one of these tasks, and view it in a dual way -as a classifier, by providing a probability distribution over the output labels for a given input, and as a representation extractor, providing an embedding of the inputs, capturing the specific aspect of the sub-task (e.g., embedding the user's initial bias on a topic), using the module's final hidden layer. our main technical challenge is to design a learning and reasoning framework that can combine the supporting tasks and the final cms prediction. we explore three approaches for this purpose: intuitively, both the ensemble-based and the modular approaches exploit the sub-tasks modules. the first looks at the modules' outputs, while the second uses the modules as representation extractors, incorporated into downstream tasks representations, and adapted during the training process. in the rest of this section we explain the different choices. we begin with the basic modules for each task, trained in an end-to-end fashion (sec. 5.1). then, in sec. 5.2, we explain our hierarchical modular approach. finally, we explain the multi-task (sec. 5.3) and ensemble-based (sec. 5.4) approaches. ## experimental design we run each setting using a 10-fold cross validation. for each fold the data was separated by debates, i.e. all votes of the same debate are associated with the same fold. in each run, one fold was used for testing and the rest for validation and training. we randomly chose 15% (20% for arguments) of the training as validation data keeping votes of the same debates together. we use the adam optimizer with a 256 batch size and the sigmoid activation function. due to the high class imbalance, we balance the weight of the classes in the objective function. for the arguments, after, before and changemystance modules we use a learning rate of 0.0001. training stops if there is no improvement after 25 epochs or 200 epochs have passed. agreement uses a learning rate of 0.001 and stops after 50 epochs of no improvement. raw inputs the representation of each debate stance is the bert encoding of the first 510 characters (9216 features). this representation decision helped limit the computational burden, however it can potentially lead to information loss, as in some cases the combined arguments of each stance can be longer, which was the case in 50% of the debates. we ran two experiments to validate this design decision, comparing the e2e cms model trained over the truncated text to two systems trained over the full text. the first was a bag of words of the top 20,000 frequent unigrams, resulting in a 6% in +f1 score drop. the second model used a separate bert encoding of each debate round, and the full stance representation was created by averaging these vectors. this representation imposed a higher computational cost, however it did not lead to a statistically significant result difference compared to the truncated version. the user profile u prof ile has 158 features by putting together demographic information and big issues. the randomly generated user embedding u emb is 100d (50d for arguments). the user summary u summary was used only as part of the agreement module and it has the same number of features as the bert encoding. we test statistical significance "*" with p -@xmath5.01 over the closest simplified version. for example: "+ u emb " is tested against "+ u profile ". ## conclusions and future work in this paper we study the problem of identifying whether debate arguments will convince their reader to change their minds. we focus on a highly challenging version of this task in which only 6% of the users actually change their mind, requiring models that can capture the interaction between users, their biases and relevant arguments for them. these settings, while challenging, are natural -we often would like to predict the impact of different arguments on different demographics, allowing policy makers to adapt their arguments and help people make better choices. we propose a hierarchical modular learning approach, which relies on decomposing the stance-change task into different aspects, such as characterizing the users biases and the quality of the arguments made. we describe a general formulation for composing these modules recursively. our approach can be considered as a form of scaffolded learning, in which we construct increasingly complex task representations in a hierarchical way, by using the previously learned modules to construct a higher level representation, or scaffolding, supporting the new learning task. our experiments show the strength of this approach, raising directions for future research on leveraging modular architectures on a larger scale, to capture different forms of users' behaviors and characterize the arguments in more nuanced ways.