A dam-Smith at S em E val-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models (2023)

Abstract: This paper presents the best-performing approach alias “Adam Smith” for the SemEval-2023 Task 4: “Identification of Human Values behind Arguments”. The goal of the task was to create systems that automatically identify the values within textual arguments. We train transformer-based models until they reach their loss minimum or f1-score maximum. Ensembling the models by selecting one global decision threshold that maximizes the f1-score leads to the best-performing system in the competition. Ensembling based on stacking with logistic regressions shows the best performance on an additional dataset provided to evaluate the robustness (“Nahj al-Balagha”). Apart from outlining the submitted system, we demonstrate that the use of the large ensemble model is not necessary and that the system size can be significantly reduced.

## introduction our system uses an ensemble of transformerbased models that are either trained until they reach their loss minimum or the f1-score maximum. to ensemble the models we average the individual predictions and calculate a decision threshold for the final system on a separate "leave-out-dataset". this model achieves the best performance in the competition ("main" dataset). each team was allowed to submit up to four systems. ensembling the predictions by using stacking with logistic regressions leads to the system with the best performance on the additional "nahj al-balagha" dataset. in this paper, we describe the best-performing system on the "main" dataset and briefly outline the ideas behind the other submitted systems. the system can be accessed through a web-based interface that is available online foot_0 . furthermore, a docker container, models, and code are open source and publicly accessible (appendix a). ## system overview this section is dedicated to precisely describing the best-performing submission as a baseline for future research and further development. the proposed system is an ensemble of 12 individual models. figure 1 provides an overview of the inference pipeline of the final system. we briefly outline the process of making a prediction and subsequently describe each of the individual steps in detail. to make a prediction the following steps are performed: ## experimental setup the final system was trained on the data provided by the task organizers (training + validation set) except for a "leave-out-dataset" of 300 samples. during the different cross-validation runs a validation set of 500 samples is taken from the training data. we use a linear learning rate schedule and early stopping. we stop the training process if the validation loss or f1-score does not improve in 3 consecutive evaluation steps. especially the parameter for the learning rate schedule (total_training_steps) was manually optimized because it defines the speed at which the learning rate decays and is therefore crucial for the learning process. the hyperparameters for pretraining and finetuning, used model versions as well as links to the code and a docker container reproducing the results can be found in appendix a. during the competition, we used an internal test dataset of 500 samples to create an internal "leaderboard" and choose the final systems to submit in the competition. the models are implemented with pytorch-lightning and are trained on an nvidia tesla t4 gpu. ## results each team was allowed to make up to four submissions ( en-thres-lod: the system described in this paper and the best-performing system in the competition. the ensemble threshold is calculated on a "leave-out-dataset" ("en-thres-lod"), and the model achieves an f1-score of 0.56 ( en-log-reg: a system that uses stacking with logistic regressions as an ensemble method (section 3.3). it has an f1-score of 0.54 on the "main" dataset and was the best-performing system with an f1-score of 0.40 on the "nahj al-balagha" dataset. en-thres-train: the same system architecture as en-thres-lod. instead of calculating the threshold on the "leave-out-dataset", the entire dataset was used for training and the optimal threshold was calculated on the training data. en-silver-labels: the system uses a selftraining approach (appendix b.1). a first version of a system creates additional training data ("silver labels") for arguments from the ibm-30k dataset. the model architecture is the same as "en-thres-lod". the size of the training data is then increased to 140% by adding the "silver-label" data. we followed a similar approach as in @xcite . the performance during training seems to slightly improve (figure b .1), but when ensembled the system was outperformed by the other architectures. all of our four submitted systems rank high in the competition on the "main" dataset and the "nahj al-balagha". on these datasets, our systems outperform the bert baseline by a large margin. however, there is a different picture for the "new york times" dataset. with a maximum score of 0.27, the systems are only slightly better than the bert baseline and are outperformed by other systems by a large margin (0.34 as "best approach" in the scores reported in ## conclusion we have presented the best-performing system for the task of automatically detecting human values in arguments. our system ensembles 12 models that have been either optimized for loss minimization or f1-score maximization. as an ensemble method, we choose one global decision threshold for all labels. the threshold maximizes the f1-score for a "leave-out-dataset". this approach outperforms stacking as an ensemble method, where for each label a logistic regression is trained. even though our systems show the best performances on the "main" and "nahj al-balagha" datasets, they are outperformed by other approaches on the "new york times" dataset. in the ablation study, we show that such a large ensemble is not necessary. in fact, an ensemble of only 3 models shows better performance and robustness while decreasing the system's memory requirements significantly. for future work, an analysis of this phenomenon could be considered. does the reduction of ensemble size lead to a more robust system and what are the counter-productive elements in the larger ensemble? furthermore, we have only manually combined a few different models into an ensemble. hence, it would be interesting to see whether a systematic selection of different approaches within an ensemble could further boost performance.