Andronicus of Rhodes at S em E val-2023 Task 4: Transformer-Based Human Value Detection Using Four Different Neural Network Architectures (2023)

Abstract: This paper presents our participation to the “Human Value Detection shared task (Kiesel et al., 2023), as “Andronicus of Rhodes. We describe the approaches behind each entry in the official evaluation, along with the motivation behind each approach. Our best-performing approach has been based on BERT large, with 4 classification heads, implementing two different classification approaches (with different activation and loss functions), and two different partitioning of the training data, to handle class imbalance. Classification is performed through majority voting. The proposed approach outperforms the BERT baseline, ranking in the upper half of the competition.

## introduction the "human value detection" shared task @xcite relates to the identification of values that can be associated with an argument. in the context of the shared task, a single argument is represented by a single premise, a conclusion, and information on what the stance of the premise is towards the conclusion. each argument can be classified into one or more value categories, among the 20 value categories, based on @xcite . the shared task covers a single language, english, providing 5393 examples for training, 1896 examples for validation, while participating systems are evaluated on 1576 test examples @xcite . with the ever-increasing ubiquity of artificial intelligence in real-world usage scenarios, the awareness of the so-called alignment problem @xcite is increasing, as is the need to develop novel strategies to measure and verify the alignment of machine learning (ml)/artificial intelligence (ai) systems @xcite with human values. however, high-quality datasets and novel (moral) value recognition methods are needed to further develop this area. some existing initiatives presenting datasets for human value detection are based: either on specific types of texts, such as the semeval argument annotation task @xcite , or on broader over-arching studies of values and their analysis in different spatiotemporal contexts, such as the vast project approach @xcite . nevertheless, since human value recognition in texts is a problem that can be challenging even for humans, let alone for machines @xcite , despite the existing datasets, additional methods need to be developed to detect values in various scenarios. in order to approach the challenge, several different natural language processing (nlp) and ml/ai approaches have been previously proposed and tested @xcite @xcite . several modern and well-performing techniques for text classification (of shorter text sequences) are based on artificial neural networks (ann), commonly based on lstm, gru, cnns, and rnns @xcite , and frequently using the so-called transformer architectures @xcite , and attention @xcite . to that effect, we focused on building upon the previously described successful approaches. specifically, we designed different ann architectures building upon bert @xcite and roberta @xcite pre-trained models for the english language and fine-tuning them using the semeval 2023 human values dataset @xcite . additionally, we utilized approaches that have proven successful in short text feature extraction and classification, such as siamese networks @xcite , and training the networks by treating the under-represented classes separately @xcite by providing a specialised classification head. ## background as mentioned in the introduction, our approach was built upon bert and roberta models, with the models incorporated into a wider ann with three different approaches (additional details are provided in the system overview): • a single classifier with 1 classification head outputting 20 scalars, each of which corresponds to a value category; the sigmoidf1 loss function @xcite was used to train this classifier. despite the fact that we have not observed a significant correlation among the value categories, we opted to design a multi-label approach. initially seeking to exploit multi-task learning (using stance prediction between premise and conclusion as a secondary task), we ended up using slightly different ways in performing the same multi-label classification problem: a classification head that involves a sigmoid layer (a typical approach for multi-label classification), and a classification head that involves a softmax layer (typically used in multi-class classification), following @xcite . the two classification heads are combined through voting, to produce the final classification, selecting only the value categories that both heads agree. returning to the problem of class imbalance, we tried to "sub-sample" the dataset, through the removal of associations with the majority classes from training examples that are associated with more than 3 value categories, including any majority class. the motivation behind this removal of assignments was the hypothesis that there may have been an annotation bias towards some values (like "universalism: concern", "security: personal", and "security: societal"), either due to annotators' personal biases, or due to the presence of values in the selected arguments. despite the fact that the transformer block ... transformer block [cls] p1 p2 ... p k s1 s2 ... s l c1 c2 ... cm [sep] linear (out size: 768) tanh sigmoid linear (out size: 20) dropout a softmax linear (out size: 20) dropout b sigmoid linear (out size: 20) dropout c softmax linear (out size: 20) dropout d positive weighting subsampling: minority classes only three majority classes were significantly reduced (as shown in finally, we designed an approach which combines the idea of the end-to-end ensemble and simultaneous upsampling/subsampling, with a siamese network architecture (figure 4 ), exploiting different pre-trained models, bert and roberta. this approach constitutes our fourth entry (entry "2023-01-30-16-39-54"), which is not part of the official evaluation, as it was submitted after the submission deadline (but before the evaluation results were made public). 1 minority classes have been defined as the value categories that have a frequency lower than 850 in the combined training and validation datasets. ## system overview overall, all approaches were based on the huggingface library @xcite , utilising a pytorch backend @xcite , and a custom dataloader that was created for the task, described in more detail in the experimental setup foot_0 . ## experimental setup the three different models also used different setups, and full details are available in github foot_1 , containing the full source code used for the task. ## results our best approach ranked in the upper half of the competition, and achieved a result above the bert baseline of the shared task. as seen in ## conclusion our approach ranks in the upper half of the competition. we have tested a few different approaches, 4 we have used data augmentation and pre-trained models. an increased performance over baseline was achieved by combining multi-task learning with heads that: a) implement the classification task differently, and b) are trained on different partitions of the data that simultaneously upsample/subsample the training examples. since different models showed different performance characteristics for some of the classes, an expansion of the idea would be the use of ensemblebased methods, potentially combined with automated ml approaches, to further improve the results.