Stance Prediction for Contemporary Issues: Data and Experiments (2020)

Abstract: We investigate whether pre-trained bidirectional transformers with sentiment and emotion information improve stance detection in long discussions of contemporary issues. As a part of this work, we create a novel stance detection dataset covering 419 different controversial issues and their related pros and cons collected by procon.org in nonpartisan format. Experimental results show that a shallow recurrent neural network with sentiment or emotion information can reach competitive results compared to fine-tuned BERT with 20x fewer parameters. We also use a simple approach that explains which input phrases contribute to stance detection.

## introduction stance detection identifies whether an opinion is in favor of an idea or opposes it. it has a tight connection with sentiment analysis; however, stance detection usually investigates the two-sided relationship between an opinion and a question. for example, 'should abortion be legal?' or 'is human activity primarily responsible for global climate change?' contemporary debatable issues, even though non-political, usually carry some political weight and controversy. for example, legislators may allow soda vending machines in our school or consider obesity as a health issue that directly impacts soda manufacturers and insurance companies respectively. on a larger scale, an issue such as climate change is being discussed in us presidential debates constantly. meanwhile, information about these issues is mostly one-sided and provided by left or right partisan resources. such information forms public beliefs, has persuasive power, and promotes confirmation bias @xcite , the humans' tendency to search for the information which confirms their existing beliefs foot_0 . confirmation bias permits internet debates and promote discrimination, misinformation, and hate speech, all of which are emerging problems in user posts of social media platforms. although there are many attempts to automatic identification and removal of such contents from online platforms, the need for accessing bi-partisan information that cultivates critical thinking and avoids confirmation bias remains. in this regard, a few web sources, such as procon.org, present information in a non-partisan format and being used as a resource for improving critical thinking in educational training by teachers 2 . here, we aim to improve such resources by automatic stance detection of pro or con-perspectives regarding a debatable issue. we extend our previous work @xcite by creating a new dataset from procon.org with 419 distinct issues and their two-sided perspectives annotated by its experts foot_2 . then, we leverage external knowledge to identify the stance of a perspective towards an issue that is mainly represented in the form of a question. the latest progress in pre-trained language models (howard and ruder, 2018) and transformers @xcite allows one to create general models with less amount of effort for task-specific text classification. in this work, we show that bidirectional transformers can produce competitive results even without fine-tuning by leveraging auxiliary sentiment and emotion information @xcite . experimental results show the effectiveness of our model and its remarkable performance. the model has a signif-icantly smaller size compared to the bert-base model. the main contributions of this work are as following: ## related works we group stance detection methods based on underlying data and approaches as follows: current approaches on stance detection use different types of linguistic features, including word/character n-grams, dependency parse trees, and lexicons @xcite @xcite . there are also end-to-end neural network approaches that learn topics and opinions independently while joining them with memory networks @xcite , bidirectional conditional lstm @xcite , or neural attention @xcite . there are also some neural network approaches that leverage lexical features @xcite . a consistency constraint is proposed to jointly model the topic and opinion using bert architecture @xcite . it trains the whole massive network for label prediction. none of these approaches incorporate bidirectional transformers with sentiment and emotion in a shallow neural network as we propose in this paper. additionally, our focus is to find the stance of 100-200 words long discussions, which are commonly present in nonpartisan format. ## dataset we collect data from procon.org, a non-profit organization that presents opinions on controversial issues in a nonpartisan format. issues (questions) and their related responses are professionally researched from different online platforms by its experts. the dataset covers 419 different detailed issues ranging from politics to sport and healthcare. the dataset instances are pairs of issues, in the form of questions, and their corresponding perspectives from proponents and opponents. each perspective is either a pro or a con with 100-200 words that supports its claim with compelling arguments. ## model utilizing pre-trained models has been widely popular in machine translation and various text classification tasks. prior efforts were hindered by the lack of labeled data @xcite . with the growth of successful pre-trained models, a model fine-tuned on a small portion of data can compete with models trained on 10× more training data without pre-training @xcite . recently, transformer models trained on both directions of language simultaneously, such as bert @xcite and xlnet @xcite , health and medicine 1-should euthanasia or physician-assisted suicide be legal? 2-is vaping with e-cigarettes safe? education 1-should parents or other adults be able to ban books from schools and libraries? 2-should public college be tuition-free? politics 1-should recreational marijuana be legal? 2-should more gun control laws be enacted? science and technology 1-is cell phone radiation safe? 2-should net neutrality be restored? entertainment and sports 1-are social networking sites good for our society? 2-do violent video games contribute to youth violence? here, we compute sentence-wise sentiment using vader to let the model learn the flow of sentiment across the opinion. so, each to- for an input sequence with t tokens, h bert t is the hidden state of the last bert-base layer corresponding to the input token at time t, e snt t is sentiment embedding of the token, [;] denotes concatenation operator, @xmath0, and w, b are parameters of a fully connected layer. recall that our task is to identify the stance of long opinions; so, important information towards the final stance might be anywhere in the opinion. because of that, we collect such information from the recurrent hidden states of all input tokens using max and averagepooling. max-pooling returns a vector with maximum weights across all hidden states of input tokens for each dimension. in this way, the input tokens with higher weights will be engaged for stance prediction. aside from that, the last hidden state of the recurrent network (z t ) is concatenated with the pooled information (u). finally, a dense layer transforms vector u into the class dimension. figure 1 shows the model architecture. we refer to this model as vader-sent-gru and report the experimental results in section 6. here, we use unidirectional --→ gru as it shows more stable results in our pilot experiments. ## experiments in this section, we describe the corresponding baselines followed by the training setup. ## results and discussion experimental results are provided in among the models with pairs of input, vader-sent-gru gains the highest recall and f1 score. it indicates that the external knowledge gained from a massive corpus, fine-tuned on 20× fewer parameters and enriched with sentiment information can compete with the original architecture (75.92 vs 76.90, @xmath1.0001 ). as the model is significantly smaller, it trains faster and needs fewer resources for training. nrc-emotion-gru, highlighted in gray, achieves the second-highest f1 score among the models; it reveals that adding emotion information improves stance detection (75.92 vs 76.51, spective supports the claim and dissimilar when it opposes the claim. this method works for claims and perspectives of the perspectrum dataset where the two input components are short sentences with 5 -10 words long. however, in our dataset, we have a question and its perspective that spans multiple sentences. so, forcing the model to make the bert representations of [question] and [perspective; question] similar or dissimilar, according to the stance, harms the model training. because the input components have different characteristics utilizing this method results in lower performance than the base model (bert). next, we present some experiments to better understand the model's units. ## conclusion we propose a model that leverages bert representation with sentiment or emotion information for stance detection. we create a new dataset for the perspectives that are as long as a paragraph covering a wide variety of contemporary topics. the experiments on our benchmark dataset highlight the effect of emotion and sentiment in stance prediction. the model can improve bert base performance with significantly fewer parameters. we also explain the contribution of essential phrases of perspectives in detecting their stance using maxpooling operation.