Content-based Stance Classification of Tweets about the 2020 I talian Constitutional Referendum (2021)

Abstract: On September 2020 a constitutional referendum was held in Italy. In this work we collect a dataset of 1.2M tweets related to this event, with particular interest to the textual content shared, and we design a hashtag-based semi-automatic approach to label them as Supporters or Against the referendum. We use the labelled dataset to train a classifier based on transformers, unsupervisedly pre-trained on Italian corpora. Our model generalizes well on tweets that cannot be labeled by the hashtag-based approach. We check that no length-, lexicon- and sentiment-biases are present to affect the performance of the classifier. Finally, we discuss the discrepancy between the magnitudes of tweets expressing a specific stance, obtained using both the hashtag-based approach and our trained classifier, and the real outcome of the referendum: the referendum was approved by 70% of the voters, while the number of tweets against the referendum is four times greater than the number of tweets supporting it. We conclude that the Italian referendum was an example of event where the minority was very loud on social media, highly influencing the perception of the event. Analyzing only the activity on social media is dangerous and can lead to extremely wrong forecasts.

## introduction on september 20 and 21, 2020, a constitutional referendum was held in italy to reduce the number of parliamentarians (from 630 to 400). 69.96% of the voters approved it, with a voter turnout of about 51% foot_0 . since the main italian political parties supported the referendum, at first the outcome was obvious, but, through a huge activity on social media, opposers unsuccessfully tried to overturn the result. the referendum was a confirmatory referendum: voters were asked to approve a law. thus, we refer to people that voted "yes", agreeing with the introduction of the new law that reduces the number of parliamentarians, as supporters, and we refer to people that voted "no", against the introduction of the new law, as opposers. since an always greater number of people share their thoughts online, social network analysis helps understanding the causes and forecasting the outcomes of political events, in parallel with already widely used approaches such as surveys and pools @xcite . like surveys, selection biases are hard to remove. social media users and citizens have different demographic distributions, resulting in under-represented categories of people (e.g., elderly people) @xcite foot_1 . moreover, social media are also populated by bots, softwares that run accounts and automatically share content, introducing noise and bias in the collected data @xcite . these accounts are not run by real people and the data shared by them should not be included to perform analysis and statistics. however, a big advantage of the analysis of social media data is the higher magnitude of available data, easy to collect and process. it is often less expensive to collect content from social media than using classical approaches. in this study we collect and analyze twitter data about the italian referendum in 2020. our contributions can be summarized as follows: ## related works numerous published works correlate social media data with elections or referendums. the main and most studied recent event is the brexit referendum, largely investigated from many different points of view @xcite @xcite @xcite , but many other political events have been analyzed from a social media perspective @xcite @xcite @xcite . a general approach to quantify controversy in social media has been proposed by @xcite , designing a graph-based approach using solely on the underneath social graphs. this approach is language independent, relying solely on the social structure of communities of users, but computational expensive. another approach has been proposed, that includes the content of texts to make more precise and fast computations @xcite . we investigate this event from a content-based stance detection perspective @xcite , analyzing only user-generated content to detect the inclination about the referendum in italy. there are few works about stance detection with non-english tweets @xcite . @xcite collect a similar dataset for the italian referendum in 2016. they tackle the stance detection task by adding to simple nlp approaches, such as bag of hashtags, bag of mentions or bag of replies, network based features obtained by clustering the retweet/quote/reply networks with louvain modularity algorithm. they also analyze the datasets from a diachronic perspective by splitting the time window into four sections based on the dates of referendum-related events. other works focus on the italian political situation of twitter users with content-based approaches @xcite @xcite . they collect tweets shared by politicians and their followers, and train accurate classifiers that predict the political inclination of users, without considering the social interactions: the content shared contains enough information to successfully perform classification of political inclination. similar tasks have been proposed at se-meval 2016 @xcite , ibereval 2017 @xcite , ibereval 2018 @xcite and finally at evalita 2020 @xcite , where teams were challenged to detect stances of manually labeled italian tweets about the sardine movement. we remark the difficulty of such tasks by looking at the performance of the best team @xcite , that fine-tuned an italian pre-trained bert model @xcite and augmented the data with results from three auxiliary tasks. a comparative study @xcite shows that for stance-detection datasets of english texts from web and social media, bert model achieves the best performance, but there is still much room for improvements. ## data collection, description and labeling the dataset is collected from twitter foot_3 , a microblogging platform widely used to discuss trending topics, whose official api allows a fast and comprehensive implementation. on twitter, users share tweets, small texts (up to 280 characters) that can be enriched with images, videos or urls. other users can quote (or retweet) another tweet by sharing it with (or without) a personal comment. a user can also follow other users to get a notification when they tweet (retweet or quote), and can be followed by other users. we query data about the referendum held in italy in september 2020 by searching italian tweets, containing at least one of the keywords reported in the keywords are refined and validated iteratively. starting from three keywords (referendum, iovotosì -ivoteyes, iovotono -ivoteno), we inspect the most frequent hashtags and, if related to the topic, we add them to the query. in figure 1 we show the most used hashtags in our complete dataset. many frequent hashtags have no clear and safe connection with the referendum, thus we do not select them as keywords during the collection step, such surnames of politicians ("dimaio") and political parties ("m5s"). ## data analysis in this section we describe the cleaning process, the stance classifiers and their results on the collected dataset. ## biases analysis in this section we inspect three common biases that often affect the accuracies of classifiers: length of texts, lexicon and sentiment. ## conclusion thanks to the last referendum in italy, we collected a big italian stance detection user-generated dataset. the dataset consists in 1.2m tweets, of which 85k are cleaned and labeled as supporters or against the referendum. the designed hashtag-based semiautomatic labeling approach allows us to train an accurate classifier that generalizes well also on tweets that do not contain gold hashtags. we considered three common dataset biases (lengthbias, lexicon-bias and sentiment-bias), confirming no significant dangers. finally, we investigated the discrepancy between the fraction of collected tweets labeled by stance and the real outcome of the referendum, observing no clues that explain this difference. based on our findings, we suggest that drawing conclusions following social media analysis should be performed carefully, and the results should be integrated with other other classical approaches such as surveys. in future works, we aim to build a three-classes stance classifier, that can also predict neutral texts, since we observed big magnitudes of data that does not explicitly state a stance. we will also move the focus from tweets to users, detecting their inclination by looking at the history of shared tweets. we believe that the investigation of users that changed stance during the time window could help us understand how people opinions are influenced by social media. finally, we observe that our classifier do not generalize well on other italian stance-detection data sets, due to the high specificity of the task: the model learned the debate about the 2020 italian constitutional referendum and its actors' inclination, but the knowledge obtained is not adequate to perform zero-shot transfer to other data sets. however, we plan to investigate if we can obtain boosts of performances in a multi-task and multi-source context, training a model on multiple similar tasks and data at the same time.