IUST at C limate A ctivism 2024: Towards Optimal Stance Detection: A Systematic Study of Architectural Choices and Data Cleaning Techniques (2024)

Abstract: This work presents a systematic search of various model architecture configurations and data cleaning methods. The study evaluates the impact of data cleaning methods on the obtained results. Additionally, we demonstrate that a combination of CNN and Encoder-only models such as BERTweet outperforms FNNs. Moreover, by utilizing data augmentation, we are able to overcome the challenge of data imbalance.

## introduction climate change stands as one of the most critical challenges of our time, impacting ecosystems, economies, and communities worldwide. at the same time, understanding the public stance towards this pivotal issue is increasingly vital. leveraging nlp techniques to gauge public stance on climate change, especially from twitter data, provides an innovative means to comprehend diverse perspectives and sentiments in real time. to advance research in this domain, the climateactivism 2024 shared task 1 proposes three sub-tasks focused on stance and hate event detection @xcite . sub-task c is about stance detection (also known as stance classification) which is a problem related to social media analysis, and natural language processing, which aims to determine the position of a person from a piece of text they produce, towards a target (a concept, idea, event, etc.) either explicitly specified in the text or implied only @xcite ). our best approach utilizes a combination of @xcite and bertweet to capture both local and global context information in the input text with weighted cross entropy as loss function. our experiments show that a combination of cnn and bertweet outperforms feedforward neural networks (fnns) in stance detection on climate change related tweets. we also demonstrate that data augmentation can address the challenge of data imbalance, resulting improvements in model performance. we also experiment with different data cleaning methods. moreover, the best results in the data cleaning type are achieved by removing urls and usernames, and all experiments of this method have yielded better results compared to other data cleaning methods. code and results are publicly available on https://github.com/ghazalehmahmoodi/climate_activism_stance_detection. ## data the sub-task c (stance detection) dataset is part of the multi-aspect twitter dataset @xcite . the data was collected from tweets posted between january 1, 2022, and december 30, 2022. the selection criteria involved hashtags such as #climatecrisis, #climatechange, #clima-teemergency, #climatetalk, #globalwarming, as well as activist-oriented hashtags like #fridaysfor-future, #climatestrike, etc. the dataset distribution is illustrated in ## methodology we proposed a model comprising four modules, and to determine the most suitable parameters for each module, we conducted numerous experiments with various configurations, seeking the optimal values within the defined search space ( ## experiments and results to evaluate the results, we used the marco f1-score as the main metric and also reported precision, recall, and accuracy. the hardware used in experiments is a gpu.1080ti.xlarge with 31.3gb ram. each training epoch lasts 2-5 minutes on average. in section 2.1, we introduced seven modes for data cleaning. experiments are repeated for the mode without or with data augmentation. therefore, we tested 14 configurations in total, including 7 modes for data cleaning and 2 modes for input data. for each configuration, we selected model the experimental results indicate that the cleaning method, which removes urls and usernames (c4), performs better compared to other methods. the complete cleaning and original text methods, on the other hand, yielded weaker results than other approaches. additionally, it can be said that maintaining hashtags and not converting to lowercase is a better cleaning approach because sometimes writing all letters in capital letters indicates intensity of anger or opposition. furthermore, in general, bert embeddings perform better in complete cleaning, while roberta and xlm-roberta models are more commonly used in other cleaning methods and yield better results and the best result is obtained with bertweet. regarding the classifier type, usually a cnn with 4-5 last layers achieves better results. evidence suggests that the defined ccn architecture, due to its use of different filters sizes and consideration of neighborhoods, has been able to achieve better results compared to fnn. additionally, typically, roberta and xlm-roberta embeddings are used with cnn, while bert is paired with fnn for better performance. analyzing the experiments as a whole, it can be concluded that the best results were obtained by optimizing sgd and using weighted cross entropy as loss function. comparison of our results and the baseline illustrate in parameter value epoch 8 batch size 4 dropout 0.5 learning rate 0.007903 learning schedule linear schedule with warmup embedding bertweet classifier cnn n_last_layer 5 optimizer sgd loss function weighted cross entropy to determine the impact of data cleaning on the results obtained, we repeated experiments with the best configuration (as shown in ## error analysis by analyzing the model errors, it can be concluded that as expected, the model struggles with detecting the oppose class. in addition to the low number of data points in this class, the presence of sarcasm and irony in the data makes it harder for the model to fully comprehend the situation and make accurate predictions. it is evident that in parts of the text where there is sarcasm, the probability of model error significantly increases. consider the tweet #fridaysforfuture #climatechange #extinctionrebellion #globalwarming what are we saving?. since some of the hashtags are used to collect data, they are present in all three classes. ## conclusion this work involved a systematic exploration of model architecture and data cleaning methods. we find that the optimal configuration combining bertweet and cnn with weighted cross entropy and sgd, along with data augmentation, led to achieving an impressive macro f1-score of 0.7447.