UMUT eam at S em E val-2023 Task 11: Ensemble Learning applied to Binary Supervised Classifiers with disagreements (2023)

Abstract: This paper describes the participation of the UMUTeam in the Learning With Disagreements (Le-Wi-Di) shared task proposed at SemEval 2023, which objective is the development of supervised automatic classifiers that consider, during training, the agreements and disagreements among the annotators of the datasets. Specifically, this edition includes a multilingual dataset. Our proposal is grounded on the development of ensemble learning classifiers that combine the outputs of several Large Language Models. Our proposal ranked position 18 of a total of 30 participants. However, our proposal did not incorporate the information about the disagreements. In contrast, we compare the performance of building several classifiers for each dataset separately with a merged dataset.

## introduction this manuscript summarizes the participation of the umuteam in the second edition of the learning with disagreements (le-wi-di) shared task @xcite , proposed in semeval 2023. le-wi-di focuses on the development of supervised automatic classifiers in which the training process considers the agreement and disagreement of the annotators. for this purpose, in this edition of le-wi-di the organizers have released a multilingual dataset composed of four normalized datasets focused on entirely subjective tasks. according to the organizers, one of the underlying objectives of this task is to develop a unified testing framework for learning from disagreement. our proposal to solve the le-wi-di shared task is based on the training of several deep learning classifiers and then combine their outputs using ensemble learning. to this end, we evaluate several large language models (llms), and we also consider the usage of linguistic features (lfs) from umutextstats @xcite . besides, we evaluate the training of a unique classifier for all datasets, or the strategy of training supervised classifiers specialized in each of the datasets. however, our major limitation is that we could not incorporate to the training of our models the information about disagreement. our participation outperformed the baseline and other participants, even though it does not incorporate disagreement. accordingly, our proposal could be a more real baseline than the one proposed by the organizers of the task. in addition, we have created a github foot_0 repository with some resources concerning our participation. ## background one of the key challenges of this edition of le-wi-di is that it is not focused on a specific topic. instead of this, the organizers of the task proposed different datasets, all based on binary classifications. the four datasets involved in this task are: i) the hs-brexit dataset @xcite , focused on hate-speech and abusive language on brexit. this dataset was annotated by six annotators that were a target group of three muslim immigrants in the uk, and a control group of three other persons; ii) the armis dataset @xcite , composed of tweets written in arabic, and annotated for misogyny detection. in this case, the annotators have different demographic traits; iii) the convabuse dataset @xcite , composed by 4,185 dialogues written in english. these dialogues took place between users and two conversational agents. this dataset was annotated by three or more experts in gender studies; and iv) the multidomain agreement dataset, composed by 10k english tweets concerning blm 2 , elections, and covid-19. in this case, the dataset contains annotations concerning offensiveness performed by 5 annotators. it is worth noting that this dataset was annotated with disagreement in mind. for this, almost 1/3 of the dataset has then been annotated with a 2 vs 3 annotators disagreements, and a third one of the dataset has an agreement of 1 vs 4. for each dataset, we reserve from the training split a 20% of the documents for performing a custom validation before the final submission. the dataset statistics are depicted in ## system overview the system architecture proposed by our research group for solving the le-wi-di task is depicted in figure 1 . in a nutshell, it can be described as follows. first, we clean all datasets by stripping hyperlinks, mentions and other similar stuff. second, we use a dataset splitter that returns all the proposed datasets merged or separated. this is because we evaluate two different strategies. the first one is the training of a model from all the datasets combined. this strategy is like a multi-task training. the second one is the training of separate classifiers focused on each one of the specific datasets proposed. the third stage of our pipeline is the fine-tuning of three llms based on bert and roberta architectures. once fine-tuned, we extract the sentence embeddings from each document and llm for the next step. it is worth mentioning that we evaluate another feature set based on lfs, but we eventually decided to remove it from our pipeline because it downplayed the overall results. fourth, we combine all the sentence embeddings from the fine-tuned models of each llm into a 2 black lives matters unique classifier. for this, we evaluate, on the one hand, three ensemble learning based on different averaging strategies and, on the other, a knowledge integration (ki) strategy that consists in training a new classifier from the fine-tuned sentence embeddings from each model. finally, according to the results of a custom validation split, we decided our final models for the official submission. for the data cleaning stage, as some of the documents of all datasets are written in different languages, we apply a simple cleaning pipeline that removed hyperlinks, hashtags and mentions from the datasets. the evaluated feature sets are described below: bert is a pretrained model that learns contextual word-embeddings; that is, embeddings that have different representation for a word, according to its surrounding words. next, we train a supervised classifier based on the lfs and we fine-tuned all the llms that have been evaluated. once fine-tuned, we extract the sentence embeddings from the [cls] token, in a similar fashion as sentence bert does @xcite . the fine-tuned process is conducted by a hyperparameter optimization stage, which consisted in the training of 10 models in which the weight decay, the batch size, the warmup speed, the number of epochs, and the learning rate vary using the tree of parzen estimators (tpe) @xcite . next, we combine the outputs of each model using two strategies. the first strategy is based on ki, which consists in training a new model from sentence embeddings for each llm. this new model is trained using keras. the second strategy is the usage of ensemble learning, which consists in combining the predictions of probabilities output of each model separately to generate a more robust option. the ensemble learning strategies evaluated are: ## experimental setup first, we report our results with the custom validation split. as stated before, we evaluate two main strategies. our first strategy consisted in training a unique model from all the datasets combined into a larger one. the results of these models with our custom validation set are reported in ## results the evaluation of the le-wi-di shared task is based on two metrics. the first metric is a soft metric, and it is the main evaluating metric. this metric considers how well the probability of each binary model reflects the level of agreement among annotators. the second metric is a hard metric, and it is the f1-score. the participants were allowed to send a total of 5 runs. however, me commit a mistake generating the first runs and run out of time. because of this, we could only send to the official leader board 3 runs, based on the ensemble learning models. we decided to send the results of the best supervised classifier trained from each dataset separately because we considered a better strategy based on the previous results. from the results posted in the codalab platform, we achieved a f1-score of 0.8176 using the ensemble learning strategy of the mode of the predictions, a f1-score of 0.8176 with the highest probability strategy, and a f1-score of 0.8303 averaging all the probabilities. we leave the latest submission as our official submission. as expected, our results considering the f1score metric are better. in fact, we achieve the position number 13 in the leader board. our results considering the f1-score metric for each dataset are reported in next, our results by dataset are reported in ## conclusion in these working notes, we have described the participation of the umuteam in the le-wi-di shared task, in which we have ranked position 18 from a total of 29 participants plus a baseline. our participation is grounded on the development of several ensemble learning classification models based on three llms and lfs. we are happy with our participation, but we are aware that we achieved modest results. besides, we could not have included information concerning the agreement among the annotators. the differences in our ranking among the soft and hard metrics give value to the importance of considering the disagreements of the annotators in machine-learning experiments, as we achieve as further work, we will try to incorporate this knowledge to our pipeline.