Fine-Tuning Language Models on D utch Protest Event Tweets (2024)

Abstract: Being able to obtain timely information about an event, like a protest, becomes increasingly more relevant with the rise of affective polarisation and social unrest over the world. Nowadays, large-scale protests tend to be organised and broadcast through social media. Analysing social media platforms like X has proven to be an effective method to follow events during a protest. Thus, we trained several language models on Dutch tweets to analyse their ability to classify if a tweet expresses discontent, considering these tweets may contain practical information about a protest. Our results show that models pre-trained on Twitter data, including Bernice and TwHIN-BERT, outperform models that are not. Additionally, the results showed that Sentence Transformers is a promising model. The added value of oversampling is greater for models that were not trained on Twitter data. In line with previous work, pre-processing the data did not help a transformer language model to make better predictions.

## introduction the number of protests across the globe has grown in the last decade (e.g. @xcite ).1 public safety can be threatened at these protests when riots can break out. for example, trump supporters attacked the united states capitol in washington, d.c. on january 6, 2021 @xcite . state property was destroyed (repairs exceeding $1.5 million (united states attorney's office, 2021)) and several law enforcement officers lost their lives during the riots that followed (united states senate committee on homeland @xcite . while these examples of extreme social unrest are generally uncommon, they express a need to forecast these types of events. in the netherlands, a possibility of large-scale protests exists. an example is the curfew protests (dutch: previous protests and stricter covid rules might lead to a divided population over time due to decreased social mobility @xcite . this lack of social mobility is argued to be the source of growing discontent and polarisation in the country @xcite . it is reported that these higher levels of affective polarisation have increased in the netherlands @xcite . since polarisation might lead to more social unrest, the dutch police are interested in gaining knowledge about the emergence of protests. nowadays, it is possible to follow incidents in real time as people increasingly use social media to broadcast live events (e.g. @xcite ). as a result, x (formerly twitter) is increasingly being studied as a news reporting platform more than anything else @xcite @xcite . it is observed that disaster-related events are also being reported on x @xcite @xcite @xcite . as an example, @xcite describe the arab spring protests in 2011 as uprisings of a political nature, where social media was pointed out as having gained a more important role in these types of protests. subsequently, actors such as governments and policing agencies "aim to understand how events are reported using social media and how millions of online posts can be reduced to accurate but meaningful information that can support decision making and lead to productive action" @xcite . scholars studying social movements have argued that social networks-as established through social media-are fundamental to protest participation (e.g. @xcite ). @xcite used an event detection framework in combination with temporal, spatial and textual content features from x to detect different kinds of events, including disruptive ones and those on smaller scale. furthermore, they found that their method performs at least as well as using other terrestrial sources. in line with this, social media has been used as a way to warn people of unsafe areas and to spread awareness for disaster relief fundraising @xcite . this power that social media possesses has also been demonstrated during the haiti earthquake in january 2010, where the awareness raising resulted in 8 million u.s. dollar donations to the red cross @xcite . this suggests that understanding the dynamics of social media messaging, especially during high-impact events like protests, are key to timely decision-making. an advantage to social media analysis is that information about events can be extracted faster than official news reports publicise @xcite . however, one of the main research challenges in studying civil unrest, is the actual identification of such information in the fasts amount of data @xcite . we propose an analysis approach to recognising such information: classifying messages based on expressions of discontent. ## expression of discontent a link between discontent and collective protests is described by @xcite , where discontent is a negative feeling towards certain aspects of the world, which includes distrust in political authorities, rules, or decisions. since x is a popular way to motivate people to protest @xcite , we aim to detect expressions of discontent in tweets (see section 5.1.2 for a precise definition). we hypothesise that people expressing discontent are more likely to start protesting. this paper compares several bertje, mbert, bernice, twhin-bert and sentence transformers models fine-tuned to newly annotated datasets of dutch tweets. we include experiments with the set-fit framework and compare to a logistic regression baseline. we aim to understand how these models can identify expressions of discontent, and how well they perform on dutch protest event tweets. ## social media analysis challenges osint utilises social media analysis to gain insights into events taking place in the country. however, current social media analysis practices pose several challenges related to privacy and the availability of suitable tools. ## method the collected tweets are labelled according to the classes 'no discontent' and 'expression of discontent'. then, mbert, bertje, bernice, twhin-bert, and multilingual sentence transformer models are fine-tuned using the labelled datasets from the previous step. a logistic regression model is trained to determine baseline performance. we mainly focused on training a sentence transformers model without the setfit framework due to time and resource constraints, as the setfit framework took substantially longer to train. the models are evaluated on several metrics. the anonymised data and used code for the models are publicly available at https: //github.com/meaganium/detectingdiscontent-in-dutch-events. in summary, we evaluate whether or not there is a difference in models' performance in how well they predict a tweet's expression of discontent. ## results to evaluate the results for the eod prediction, we focused mainly on the results for the eod label, as this was the minority class for all individual datasets. see tables 3 and 4 for the prediction results. first, we ran all the models with all four dataset combined, see ## discussion in this paper, we aimed to identify how future nlp models can be improved in order to provide better predictions for social media text. our work provides an overview of several language models' performances on dutch tweets for the prediction of ## ethics statement ethical approval to conduct this study, including approval for the collection and annotation of the datasets, was acquired from the appropriate local institutional review boards and ethics committees. to minimise potential privacy issues, we excluded direct and indirect personal identifiers from the data, including names and locations. in line with the gdpr guidelines, the data has been anonymised by hashing usernames and mentions. besides the focus on dutch text, it is desirable for high-impact applications, like those used in medical practice and law enforcement, to work with models and algorithms that have low false negative rates, due to potential societal and ethical complications that arise with false positives. for example, it is unethical and socially undesirable to inaccurately label a person's social media message along the lines of 'high-risk' or 'negative'. therefore, in this work, we focused on the optimisation of the precision metric, as this indicates lower false positives. we encourage future work to put low false positive rates at the forefront in the evaluation of models' performance. furthermore, we followed the european data protection board (edpb) guidelines to assess the risks and potential impacts of the data.4 these guidelines were followed in order to minimise potential risks for individuals' freedoms, and to use the data in a lawful and transparent manner. for future work, we provide several suggestions on how to use social media data in an ethical manner. first, ethical data assessment methodologies should be used before the analysis is conducted in order to evaluate potential conflicts with (public) values and to minimise social disruption. we recommend using approaches like 'de ethische data assistent' (deda, 'the ethical data assistent') from @xcite . second, when conducting social media analysis, the focus should be on groups rather than individuals, so that privacy is ensured and the results remain 'superficial' in nature. as previously mentioned, the gdpr emphasises that monitoring and profiling is not allowed, even in the context of anticipating crimes and riots. therefore, social media analyses for research purposes should emphasise the recognition of general trends, sentiments and events instead, as presented in this paper.