G un S tance: Stance Detection for Gun Control and Gun Regulation (2024)

Abstract: The debate surrounding gun control and gun regulation in the United States has intensified in the wake of numerous mass shooting events. As perspectives on this matter vary, it becomes increasingly important to comprehend individuals’ positions. Stance detection, the task of determining an author’s position towards a proposition or target, has gained attention for its potential use in understanding public perceptions towards controversial topics and identifying the best strategies to address public concerns. In this paper, we present GunStance, a dataset of tweets pertaining to shooting events, focusing specifically on the controversial topics of “banning guns” versus “regulating guns.” The tweets in the dataset are sourced from discussions on Twitter following various shooting incidents in the United States. Amazon Mechanical Turk was used to manually annotate a subset of the tweets relevant to the targets of interest (“banning guns” and “regulating guns”) into three classes: In-Favor, Against, and Neutral. The remaining unlabeled tweets are included in the dataset to facilitate studies on semi-supervised learning (SSL) approaches that can help address the scarcity of the labeled data in stance detection tasks. Furthermore, we propose a hybrid approach that combines curriculum-based SSL and Large Language Models (LLM), and show that the proposed approach outperforms supervised, semi-supervised, and LLM-based zero-shot models in most experiments on our assembled dataset.

## introduction in the aftermath of the multitude of mass shooting events in the united states, topics regarding tighter gun regulations and control, expanding mental health services, and upgrading security in public places have been strongly debated over politi-cal campaigns on various social media platforms. people are generally on two opposing sides with respect to gun regulations and control: those who favor such regulations and those who strongly oppose them. such differences in opinions on gun regulations have prevailed in society for over four decades, resulting in very little legislative success @xcite . social media, especially twitter (now x), has been a melting pot for such debates. it would be useful to automatically analyze the intricate dynamics, challenges, and complexities surrounding gun control topics. stance detection on social media is an emerging research area in opinion mining for various applications where sentiment analysis may be suboptimal @xcite . stance detection is the task of automatically determining if an author of a text is in-favor, against, or neutral towards a proposition or target @xcite . while research on stance detection has seen significant progress in various domains, such as politics, healthcare, and finance @xcite @xcite , there remains a notable gap in analyzing stance dynamics in the context of gun control and regulations on social media, despite the importance of this topic in the contemporary society. this is mainly due to a lack of datasets labeled with respect to stance towards the controversial topics of gun control and regulation. in this work, we aim to address this need by creating gunstance, a dataset of tweets collected during mass shooting events in the united states. specifically, we focus on seven shooting events and identify two controversial, interconnected gunrelated stance targets, a stronger stance target -"banning guns" and a softer stance target -"regulating guns." we collect tweets posted during those shooting events and filter tweets potentially relevant to each of the targets considered. note that some tweets can express (either the same or different) stances towards both targets, therefore the sets of tweets filtered for the two targets are not disjoint. using amazon mechanical turk, we annotate 2,700 tweets according to the author's stance toward the "banning guns" target, and 2,800 tweets according to the author's stance toward the "regulating guns" target. in addition, the gunstance dataset includes 7,334 unlabeled tweets for the "banning guns" target and 8,824 tweets for the "regulating guns" target. the unlabeled tweets can be utilized by semi-supervised learning approaches. more details about data collection and statistics are provided in section 3. in addition to assembling the gunstance dataset consisting of labeled and unlabeled tweets for the two targets, we first establish strong baselines for this dataset by employing supervised, semi-supervised and zero-shot llm-based approaches. to push the performance on our task and increase the real-world applicability of the stance detection models, we propose a hybrid ssl/llm approach that utilizes unlabeled examples from a stream of data (i.e., posted as events unfold), while leveraging the zero-shot capabilities of llms to improve the model. critically, compared to traditional ssl, our method incurs minimal additional training costs, due to our novel approach of identifying informative unlabeled examples, and no inference costs. detailed information on our methods is provided in section 4, while the results of the models are presented and discussed in section 5. to summarize, the main contributions of this work consist in providing the research community with a dataset, a novel low-cost hybrid ssl/llm approach, and comparisons with meaningful baseline models for analyzing social media data with respect to stance towards controversial topics regarding gun regulations and control. our dataset and proposed hybrid model are designed to help uncover prevalent arguments and opinions related to these topics, and to foster a deeper understanding of the intricate dynamics surrounding gun control and regulations. by enabling the examination of diverse perspectives, our work has the potential to lead to advancements in addressing the challenges posed by gun control and regulation. ## related work in the last decade, stance detection has been an area of active research and numerous studies focused on introducing novel datasets or proposing stance detection models have been carried out. ## gunstance dataset dataset collection and filtering: our data collection process involved gathering tweets related to mass shooting incidents across different locations in the united states. using wikipedia pages that list mass shooting events in the united states in 2021 2 and 2022 3 , respectively, we identified and focused on 7 mass shooting events (3 in 2021 and 4 in the first half of 2022), each with at least 5 dead people. the events selected (shown in past tweets were retrieved in chronological order by using the twitter's v2 full-archive search 2 https://en.wikipedia.org/wiki/list_of_mass_shootings_in_ the_united_states_in_2021 3 https://en.wikipedia.org/wiki/list_of_mass_shootings_in_ the_united_states_in_2022 endpoint foot_0 . for each event, we collected tweets posted the day of the event and up to eight weeks after the event. we started the search using a basic set of seed rules specific to the location of each event. as an example, the seed rule for the robb elementary school shooting in uvalde, texas was: "((uvalde or texas) shooting) or uvaldeshooting or uvaldetexas". the seed rules for the other events are shown in table a2 in appendix a.2. the rule for an event was updated with the most frequent hashtags daily during the first week of the event and weekly during the following weeks. on each update of the rule, we identified the 15 most frequent hashtags during the update period (a day or a week), and added those hashtags as additional or terms to the current rule. in addition, we also maintained a list of terms to ignore, specifically terms that were observed among the frequent hashtags but were too general and added noise to the collection (e.g., mass, dead, breaking, update, usa, america). the terms added with each update are shown in table a3 in appendix a.2. the tweets collected were capped at a maximum of 4,500 tweets per day and 25,000 tweets per week for each event. the final dataset collected included a total of 395,485 original tweets, replies, and quoted tweets (retweets were ignored and are thus not included in this count). a preliminary analysis of the data showed that 77.48% of the tweets collected (i.e., 306,422 tweets) contained urls (e.g., links to news articles). while the tweets containing urls may express a stance towards the targets of interest in our study, given that they link to external resources (many of them related to media outlets), the stance expressed may not be the general public's stance but rather the stance of media outlets, political organizations, etc. as we aim to focus on detecting the general people's stance towards gun regulations, we removed such tweets. we also removed tweets from a list of 2,834 twitter handles corresponding to media outlets (e.g., pulpnews, nydailynews, abc, etc.). the 2,834 handles were identified based on post frequency (as media outlets have a large number of posts as shown in figure a .1 in appendix a.2) and/or the use of the "news" keyword in their username. together, we filtered out 309,395 tweets out of the total 395,485 collected tweets, leaving us with a set of 86,090 tweets. to identify tweets most relevant to the targets considered, we used sentence-bert (reimers and gurevych, 2019) to embed all tweets in the dataset, as well as the targets (expressed as "guns should be banned" versus "guns should be regulated") and ranked the tweets with respect to the targets based on cosine similarity. the similarity between these two targets themselves was 0.6980. we filtered out the least relevant tweets to these targets by applying a similarity threshold of 0.40. consequently, we obtained 11,621 tweets for "banning guns" and 13,476 tweets for "regulating guns." ## results and discussion we divide our experiments into three sets based on the data used to train and evaluate the models. experiment 1: in this experiment, we train and evaluate models independently for our two targets, "ban" and "regulate". zero-shot models do not use any training data. supervised models are trained using the combined labeled training data of all events. semi-supervised models are trained using the combined labeled training data and unlabeled data of all events. subsequently, the trained models are evaluated on the combined test data of all events. for supervised models, we find a significant performance difference in terms of f1-score between the non-transformer architecture-based models and the bertweet model. this difference can be linked to the complexity inherent within our dataset, which presents a greater challenge for the supervised models to effectively model such complexities. in contrast, bertweet was able to learn complexities present in our dataset better than the non-transformer supervised models. however, the zero-shot chatgpt outperforms the best supervised bertweet model, underscoring the power of llms, such as chatgpt, in a zero-shot setting on a very challenging task. experiment 2: this experiment follows a leaveone-event-out setting. models are trained for the "ban"/"regulate" targets independently using the training (labeled/unlabeled) data of all events except the held-out event, then tested on this event. the held-out events are "boulder", "buffalo", and "uvalde". we did not consider other events as heldout because of their relatively small size. experiment 3: in this experiment, we explore a cross-target setup, where we train the models on one stance target and test them on the other target. specifically, we train the models on the training data of the "ban" target and evaluate their performance on the test data of the "regulate" target, and vice versa. this is a particularly difficult task, as differences between stances towards the "ban" and "regulate" targets can be very nuanced. to ensure good coverage with the unlabeled data, the semisupervised models utilize the unlabeled data from both "ban" and "regulate" targets. the results of this experiment are shown in figure 3 . as we can see from the figure, aum-st +chtgpt outperforms both the supervised as well as other ssl methods for the "@xmath0. ## conclusions and future work we presented gunstance, a stance detection dataset containing tweets pertaining to the controversial topics of "banning guns" and "regulating guns" in the aftermath of various shooting events in the united states. the dataset includes manually annotated tweets together with unlabeled tweets to facilitate supervised and semi-supervised learning. we provide baseline results using established and newer supervised and ssl models for stance detection. we also utilize llms in a zero-shot setting and find that llms, such as chatgpt, are able to understand the complexities in our dataset and outperform supervised baselines. furthermore, we propose a curriculum-based semi-supervised approach that utilizes chatgpt as an external knowledge source on a small subset of hard-to-learn unlabeled examples when the ssl model is less confident, pushing both performance and computational efficiency of ssl methods, such as aum-st. the stance detection task addressed is not only timely (given the large number of shooting events), but also challenging. our work has the potential to enable researchers to use machine learning models to analyze mass shooting data, gain insights into prevalent arguments and opinions, and develop a deeper understanding of the complexities surrounding gun control and regulations. while there are other targets of interest in the context of mass shooting and gun control topics (e.g., buyback programs, background checks, mental health, etc.), we believe our work represents an important step towards understanding the general public's stance towards two extremely important and interconnected gun control targets. other related targets are left as part of future work. an evaluation of the proposed aum-st +chtgpt model on other datasets is beyond the scope of this current study, but makes another excellent topic for future work. we also plan to focus on domain adaptation and transfer learning techniques that allow stance detection models to be adapted to different domains or targets. finally, understanding similarities and differences between explicit and implicit stances in terms of llm/ssl stance detection abilities is also of interest. we make our dataset and code available on github to spur further research in this area. our work has a few limitations that should be acknowledged. first, the availability of data for certain shooting events is limited (e.g., san jose, california and tulsa, oklahoma), while other events benefit from more extensive coverage (e.g., buffalo, new york, and uvalde, texas), causing an imbalance in the amount of data that we were able to crawl for some events. second, it is important to note that our dataset is comprised of publicly available tweets, which are often generated by a subset of users who are more active and vocal on social media. therefore, the opinions expressed in these tweets may not necessarily represent the views of the majority. these limitations highlight other potential avenues for future research that could address the scarcity of data, improve the identification of implicit and explicit stances, and explore meth-ods to capture a more diverse range of perspectives on the topic of gun control and regulations. in our study, we removed tweets that contain external urls as the goal was to focus on the general public's stance as opposed to the stance of media outlets, political organizations, etc. however, it would be interesting to detect the stance of such global entities and understand how their stance influences the stance of their followers. as another limitation, we should note that the dataset that was used to pre-train chatgpt is unknown. it is not possible to assess the extent to which tweets in our dataset may have been seen by chatgpt and the effect on the performance. information about the dataset could provide light on the impressive abilities of chatgpt on the challenging task of detecting stance towards two important gun control and regulation targets.