Lon-eå at S em E val-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction (2023)

Abstract: We study the influence of different activation functions in the output layer of pre-trained transformer models for soft and hard label prediction in the learning with disagreement task. In this task, the goal is to quantify the amount of disagreement via predicting soft labels. To predict the soft labels, we use BERT-based preprocessors and encoders and vary the activation function used in the output layer, while keeping other parameters constant. The soft labels are then used for the hard label prediction. The activation functions considered are sigmoid as well as a step-function that is added to the model post-training and a sinusoidal activation function, which is introduced for the first time in this paper.

## introduction the nuances present in natural languages, such as contextual meaning or subjective interpretation of expressions are often discounted in natural language processing. in subjective tasks, such as sentiment analysis, offensive or abusive language detection, and misogyny detection, the assumption that a gold label always exists has proven to be an idealization @xcite . therefore, a growing body of research has been dedicated to analyzing the disagreement from labels provided by multiple annotators. some of the notable works on this topic are @xcite @xcite . in task 11 of semeval 2023 on learning with disagreements le-wi-di, 2 nd edition @xcite , four datasets, including three english datasets hs-brexit @xcite , convabuse @xcite , and md-agreement @xcite , and one arabic dataset, armis @xcite have been provided. all datasets include both soft and hard labels as well as some additional information. among these armis, hs-brexit, and md-agreement have a constant number of annotators throughout the dataset, whereas convabuse has a variable number of annotators. in this paper, we explore the effect of different activation functions in the output layer of a deep neural network model for the prediction of soft and hard labels. we introduce a sinusoidal activation function, which we refer to as the sinusoidal step function (ssf). we evaluate the performance of the models using the ssf, sigmoid, as well as a posttraining step function for soft-label, and therefore, hard-label prediction. the experiments indicate that the ssf activation achieves the highest f1-score for the hard-label prediction on the majority of the datasets. for the soft label prediction, the sigmoid activation obtains the best result for most datasets, however (cf. section 5) foot_0 . ## background there is a dispute on how to deal with the annotator's disagreements. whilst some suggest that we should disregard and discard instances with high disagreement as bad examples (beigman and @xcite , others argue that such instances are valuable and should be further analyzed and studied @xcite @xcite . in subjective tasks, such as of-fensive/abusive language detection and misogyny detection, introducing a framework that utilizes different views and their differences is invaluable due to the nature of such tasks, which require integrating different viewpoints. this prompts us to utilize this source of knowledge and try to find approaches for predicting soft labels which will then be used to directly infer hard labels. in the past few years transformer @xcite ) models, such as bert @xcite have been widely used for natural language processing (nlp) tasks. we adopt the bert model, trained on wikipedia and bookscorpus, for english datasets and the arabic version of bert, also known as the arabert (antoun et al., 2020), on the arabic armis dataset. ## system overview in the experiments, we focus on analyzing the performance of different activation functions with the goal of finding the best performance on the soft labels. we use three approaches for computing soft labels. among the four datasets of this competition, one of these approaches, which uses the sigmoid function as activation is applicable to all of the datasets. the other two approaches, which use the step functions we introduce, are applicable to hs-brexit, armis, and md-agreement because the number of annotators is equivalent to the number of steps, and it is fixed in these datasets. we start by describing our approaches for soft label prediction in subsection 3.1. we then discuss our results for predicting the hard label from the soft label in subsection 3.2. ## results in this section, we discuss the results obtained on each dataset. the focus of our experiments is to try different activation functions in the output layer to compare the performance of these activation functions for predicting soft labels and hard labels. our study does not involve analyzing and finding different splits for the data, and we use the splits as released by the task organizers. furthermore, we do not train the models on additional data before training them on the competition datasets as this adds more variables to the results of the experiment other than the activation functions used in this study. also, the second and third approaches are only used for hs-brexit, armis, and md-agreement datasets, as the number of annotators in the convabuse dataset is variable across different instances. the second and third approaches are only applicable to datasets where the number of annotators is a constant number across the dataset instances. we divide our discussion about the results into two parts, the results of the soft evaluation and hard evaluation. in each subsection, we present a table that summarises the results of the different approaches for the corresponding metric, on all datasets. the tables also include the results of the organizers' baseline and the best performance achieved on each metric by the participating teams in the competition. ## conclusion we studied the effect of various activation functions in the output layer of a deep neural network model, including the sinusoidal activation function ssf, introduced for the first time in this paper, on soft and hard label prediction in the learning with disagreement task at semeval 2023. the sinusoidal activation function ssf, which can be applied to any domain or dataset where the number of annotators is constant or their views are summarised into a constant number of categories, shows promising results in the hard evaluation. on two of the three datasets that ssf was applied to, i.e., armis and md-agreement), the use of ssf improves the micro f1-score by about 5.52% and 1.57%, respectively, in comparison to the widely used sigmoid activation. based on the promising performance of the ssf activation for the hard label prediction, we plan to further study the use of ssf activation on other datasets and tasks, where a constant number or class of annotators have annotated the data.