Agreement Prediction of Arguments in Cyber Argumentation for Detecting Stance Polarity and Intensity (2020)

Abstract: In online debates, users express different levels of agreement/disagreement with one another’s arguments and ideas. Often levels of agreement/disagreement are implicit in the text, and must be predicted to analyze collective opinions. Existing stance detection methods predict the polarity of a post’s stance toward a topic or post, but don’t consider the stance’s degree of intensity. We introduce a new research problem, stance polarity and intensity prediction in response relationships between posts. This problem is challenging because differences in stance intensity are often subtle and require nuanced language understanding. Cyber argumentation research has shown that incorporating both stance polarity and intensity data in online debates leads to better discussion analysis. We explore five different learning models: Ridge-M regression, Ridge-S regression, SVR-RF-R, pkudblab-PIP, and T-PAN-PIP for predicting stance polarity and intensity in argumentation. These models are evaluated using a new dataset for stance polarity and intensity prediction collected using a cyber argumentation platform. The SVR-RF-R model performs best for prediction of stance polarity with an accuracy of 70.43% and intensity with RMSE of 0.596. This work is the first to train models for predicting a post’s stance polarity and intensity in one combined value in cyber argumentation with reasonably good accuracy.

## introduction many major online and social media and networking sites, such as facebook, twitter, and wikipedia, have taken over as the new public forum for people to discuss and debate issues of national and international importance. with more participants in these debates than ever before, the volume of unstructured discourse data continues to increase, and the need for automatic processing of this data is prevalent. a critical task in processing online debates is to automatically determine the different argumentative relationships between online posts in a discussion. these relationships typically consist of a stance polarity (i.e., whether a post is supporting, opposing, or is neutral toward another post) and the degree of intensity of the stance. automatically determining these types of relationships from a given text is a goal in both stance detection and argumentation mining research. stance detection models seek to automatically determine a text's stance polarity @xcite toward another text or topic based on its textual information @xcite . likewise, argumentation mining seeks to determine the stance relationship (supporting, attacking, or neutral) between argumentation components in a text @xcite . however, in both cases, attention is only paid to the stance's polarity, while the intensity of the relationship is often ignored. some studies have tried to incorporate intensity into their predictions by expanding the number of classes to predict (strongly for, for, other, against, and strongly against); however, this expansion lowered their classification performance considerably compared classification without intensity @xcite . thus, effective incorporation of stance intensity into stance classification remains an issue. research in cyber argumentation has shown that incorporating both stance polarity and intensity information into online discussions improves the analysis of discussions and the various phenomena that arise during a debate, including opinion polarization @xcite , and identifying outlier opinions @xcite , compared to using stance polarity alone. thus, automatically identifying both the post's stance polarity and intensity, allows these powerful analytical models to be applied to unstructured debate data from platforms such as twitter, facebook, wikipedia, comment threads, and online forums. to that end, in this paper, we introduce a new research problem, stance polarity and intensity prediction in a responsive relationship between posts, which aims to predict a text's stance polarity and intensity which we combine into a single continuous agreement value. given an online post a, which is replying to another online post b, we predict the stance polarity and intensity value of a towards b using a's (and sometimes b's) textual information. the stance polarity and intensity value is a continuous value, bounded from -1.0 to +1.0, where the value's sign (positive, negative, or zero) corresponds to the text's stance polarity (favoring, opposing, or neutral) and the value's magnitude (0 to 1.0) corresponds to the text's stance intensity. stance polarity and intensity prediction encapsulates stance detection within its problem definition and is thus a more difficult problem to address. while stance polarity can be identified through specific keywords (e.g., "agree", "disagree"), the intensity is a much more fuzzy concept. the difference between strong opposition and weak opposition is often expressed through subtle word choices and conversational behaviors. thus, to accurately predict agreement intensity, a learned model must understand the nuances between word choices in the context of the discussion. we explore five machine learning models for agreement prediction, adapted from the topperforming models for stance detection: ridge-m regression, ridge-s regression, svr-rf-r, pkudblab-pip, and t-pan-pip. these models were adapted from @xcite al. (2016 2018 ) respectively. we evaluated these models on a new dataset for stance polarity and intensity prediction, collected over three empirical studies using our cyber argumentation platform, the intelligent cyber argumentation system (icas) . this dataset contains over 22,000 online arguments from over 900 users discussing four important issues. in the dataset, each argument is manually annotated by their authoring user with an agreement value. results from our empirical analysis show that the svr-rf-r ensemble model performed the best for agreement prediction, achieving an rmse score of 0.596 for stance polarity and intensity predic-tion, and an accuracy of 70% for stance detection. further analysis revealed that the models trained for stance polarity and intensity prediction often had better accuracy for stance classification (polarity only) compared to their counterpart stance detection models. this result demonstrates that the added difficulty of detecting stance intensity does not come at the expense of detecting stance polarity. to our knowledge, this is the first time that learning models can be trained to predict an online post's stance polarity and intensity simultaneously. the contributions of our work are the following: ## models for stance polarity and intensity prediction this section describes the models we applied to the stance polarity and intensity prediction problem. we applied five different models, adapted from top-performing stance classification models based on their performance and approach on the semeval 2016 stance classification twitter dataset @xcite . ## empirical dataset description the dataset was constructed from three separate empirical studies collected in fall 2017, spring 2018, and spring 2019. in each study, a class of undergraduate students in an entry-level sociology class was offered extra credit to participate in discussions in icas. each student was asked to discuss four different issues relating to the content they were covering in class. the issues were: 1) healthcare: should individuals be required by the government to have health insurance? 2) same sex adoption: should same-sex married couples be allowed to adopt children? 3) guns on campus: should students with a concealed carry permit be allowed to carry guns on campus? 4) religion and medicine: should parents who believe in healing through prayer be allowed to deny medical treatment for their child? under each issue, there were four positions (with the exception of the healthcare issue for fall 2017, which had only 3 positions) to discuss. the positions were constructed such that there was one strongly conservative position, one moderately conservative position, one moderately liberal position, and one strongly liberal position. the students were asked to post ten arguments under each issue. the combined dataset contains 22,606 total arguments from 904 different users. of those arguments, 11,802 are replying to a position, and 10,804 are replying to another argument. the average depth of a reply thread tends to be shallow, with 52% of arguments on the first level (reply to position), 44% on the second level, 3% on the third level, and 1% on the remaining levels (deepest level was 5). when a student posted an argument, they were required to annotate their argument with an agreement value. overall, argument agreement values skew positive. figure 4 displays a histogram of the agreement values for the arguments in the dataset. the annotated labels in this dataset are selflabeled, meaning that when a user replies to a post, they provide their own stance polarity and intensity label. the label is a reflection of the author's intended stance toward a post, where the post's text is a semantic description of that intention. while these label values are somewhat subjective, they are an accurate reflection of their author's agreement, which we need to capture to analyze opinions in the discussion. self-annotated datasets like this one have been used in stance detection for argumentation mining in the past (see @xcite ). ## discussion the models behaved very similarly on the agreement prediction problem, where the difference between the best performing model and the worst performing model is only 0.061. overall, the best model received an rmse of 0.596, which is reasonably good but can be improved. t-pan-pip had the worst performance, which is surprising, as it was the only model to include the parent post's information into its prediction, which should have helped improve its performance. it is possible that its architecture is unsuitable for agreement prediction; other architectures have been deployed that include a post's parent and ancestors into a stance prediction, which might be more suitable for agreement prediction. future model designs should better incorporate a post's parent information into their predictions. the difference in performance between the agreement prediction models and the classification models on the stance detection task was small and sometimes better. this demonstrates that the models learning to identify stance intensity do so without significant loss of performance in identifying stance polarity. larger gains in performance will likely require information about the post's author. some post authors will state strong levels of agreement in their statements, but annotate their argument with weaker agreement levels. for example, one author wrote, "agree completely. government should stay out of healthcare." and annotated that argument with an agreement value of +0.6. the authors were instructed on how to annotate their posts, but the annotations themselves were left to the post's author's discretion. thus including author information into our models would likely improve the stance polarity and intensity prediction results. ## conclusion we introduce a new research problem called stance polarity and intensity prediction in a responsive relationship between posts, which predicts both an online post's stance polarity and intensity value toward another post. this problem encapsulates stance detection and adds the additional difficulty of detecting subtle differences in intensity found in the text. we introduced a new large empirical dataset for agreement prediction, collected using a cyber argumentation platform. we implemented five models, adapted from top-performing stance detection models, for evaluation on the new dataset for agreement prediction. our empirical results demonstrate that the ensemble model svr-rf-r performed the best for agreement prediction and models trained for agreement prediction learn to differentiate between intensity values without degrading their performance for determining stance polarity. research into this new problem of agreement prediction will allow for a more nuanced annotation and analysis of online debate. the model was trained using a batch size of 64 and used an adam optimizer.