HAM i S o N -Generative at C limate A ctivism 2024: Stance Detection using generative large language models (2024)

Abstract: CASE in EACL 2024 proposes the shared task on Hate Speech and Stance Detection during Climate Activism. In our participation in the stance detection task, we have tested different approaches using LLMs for this classification task. We have tested a generative model using the classical seq2seq structure. Subsequently, we have considerably improved the results by replacing the last layer of these LLMs with a classifier layer. We have also studied how the performance is affected by the amount of data used in training. For this purpose, a partition of the dataset has been used and external data from posture detection tasks has been added.

## introduction case in eacl 2024 is a shared task focusing on climate activism @xcite . this task consists of three subtasks, the first two are focused on hate speech detection, a task that is important for peace and harmony in society @xcite . the last subtask consists of the posture detection of tweets on this topic. stance detection seeks to determine the author's point of view -usually in favour, against or neutral -on certain topics, using textual analysis (aldayel and magdy, 2020; @xcite . due to the large amount of information that is processed daily on social networks, stance detection has become an important task that facilitates the understanding of social and political changes @xcite . due to the large amount of information that large language models (llms) receive during their training and their good results in many benchmarks, they are being used for tasks such as posture detection in text @xcite . in which models such as chatgpt, gpt-neox @xcite , falcon 7b and 40b @xcite and llama 2 7b and 13b @xcite were used. all of them were used as sequence-to-sequence models. in this paper we will compare the performance of different llama 2 model structures for stance detection tasks. it also seeks to study how the performance is affected by the amount of data used in training. for this purpose, a partition of the dataset will be used and external data from stance detection tasks will be added. the rest of this paper is structured as follows: section 2 describes the datasets to be used along with the task to be solved. section 3 describes the methodology followed including the models, the data processing, the model inputs and the training dataset. section 4 presents the results, which will be discussed in section 5. finally, conclusions and future work are given in section 6. ## dataset and task the dataset on climate activism @xcite has been used, focusing on the subtask of stance detection. this dataset has a collection of tweets labeled according to their stance about climate activism. henceforth, we will refer to it as case. additionally, the dataset from @xcite has been employed, which is related to the stance detection task too. this dataset was used in the international workshop on semantic evaluation @xcite . it includes tweets labeled with stances about various targets such as climate change, atheism, feminism, etc. ## results this section presents the results, evaluated on the test set, of all the experiments that have been carried out. some metrics for the best performing models using the partition (1) and with the total data (2) will be shown below. figure 1 shows the normalised confusion matrix over the rows for model (1). similarly, figure 2 shows the normalised confusion matrix over the rows for model (2). part full wrong right wrong right number of systems 1 13.3 % 5.4 % 16.5 % 6.2 % 2 7.9 % 5.1 % 9.9 % 5.3 % 3 6.3 % 4.2 % 6.9 % 4.9 % 4 5.3 % 5.3 % 6.6 % 6.6 % 5 4.2 % 6.3 % 4.9 % 6.9 % 6 5.1 % 7.9 % 5.3 % 9.9 % 7 5.4 % 13.3 % 6.2 % 16.5 % 8 9.9 % 42.6 % 8.1 % 35.9 % predicted label support oppose neutral ## discussion from the results shown in regarding the addition of semeval data, if we look model by model, we see that the performance is only improved in 2 of them. the difference between adding the data at most worsens 0.0217 and at most improves 0.0163. this could be because the semeval dataset contains about 3k examples of various topics compared to 7k in case. of the semeval dataset, only 13.5% of the data was related to climate activism and 86.5% was related to another topic. this data distribution may add noise to the training. this is why the models do not specialise in stance detection on climate activism as much as using only climate activism data. however, when adding the data, there is a considerable increase in training times. as for the 4 different approaches to the llama 2 model that have been used. as the seq2seq results are the lowest, we can conclude that it is better to remove the layer that allows to obtain a text sequence and replace it by a classifier layer. in addition, although the 7b chat classifier models were the best performers, model 7b shows results with less variation when more data is added. looking at the results in comparing the confusion matrices in figure 1 and figure 2 we can see that for the support and oppose instances the model trained with the partition classifies better than the model trained with all. however, the latter classifies better the neutral instances, thus obtaining the f1 difference between both. regarding the percentages of misclassified instances per class collected in ## post-competition analysis since this is a generative model, we could use a zero-shot approach. however, using this approach llama 2 7b chat model obtained an f1 result of 0.5685. this result is somewhat higher than the baseline model proposed by the organisers, but significantly lower than the fine-tuned models. in addition, adding the semeval collection to the models caused a decrease in the performance of the models. one of the reasons could be due to the use of data not related to climate change. for this reason, the best architecture (7b chat -clf no prompt) was re-trained by adding only the semeval climate change related data. this model obtained an f1 of 0.7346, only 0.002 below the model that not use additional data. looking more closely at these two models we could see that there was only a difference of two misclassified instances. by carefully studying the structure of the semeval-2016 dataset and the case dataset, we realise that there is a temporal difference between the instances of both datasets. the case dataset contains terms such as greta thunberg or the ukrainian-russian war that semeval does not. in addition, there are hashtags such as #fridaysforfuture or #climatestrike which are movements started in 2018. therefore both datasets contain different lexical fields. ## conclusions and future work in this paper, we have reported our participation in case in the framework of eacl 2024 in the stance detection subtask. for this task we have compared the performance of several variants of llama 2 models and studied the effect of adding more data to the models. our results are significantly better than the proposed baseline model and we have found that for this classification task it is better to dispense with the seq2seq structure of llama 2 and use a classifier layer. we have also seen that adding more data tends to make the models behave worse. as lines of future work it would be interesting to make an ensemble of all the models and analyse the performance of the models by training with different percentages of the case dataset. as the smaller llama 2 model has been used, it would also be interesting to test these architectures with the larger llama 2 models, 13b and 70b. in addition, to be able to use several related collections. if they are spaced in time, more robust semantic dimensions could be studied or datasets close in time could be used.