Aristoxenus at S em E val-2023 Task 4: A Domain-Adapted Ensemble Approach to the Identification of Human Values behind Arguments (2023)

Abstract: This paper presents our system for the SemEval-2023 Task 4, which aims to identify human values behind arguments by classifying whether or not an argument draws on a specific category. Our approach leverages a second-phase pre-training method to adapt a RoBERTa Language Model (LM) and tackles the problem using a One-Versus-All strategy. Final predictions are determined by a majority voting module that combines the outputs of an ensemble of three sets of per-label models. We conducted experiments to evaluate the impact of different pre-trained LMs on the task, comparing their performance in both pre-trained and task-adapted settings. Our findings show that fine-tuning the RoBERTa LM on the task-specific dataset improves its performance, outperforming the best-performing baseline BERT approach. Overall, our approach achieved a macro-F1 score of 0.47 on the official test set, demonstrating its potential in identifying human values behind arguments.

## introduction the valueeval task aims to develop a system for automatically detecting the values expressed in natural language arguments within english texts @xcite . identifying human values is critical for gaining insights into people's behavior, evaluating content, personalizing experiences, and resolving conflicts. analyzing the values expressed in language, including beliefs, attitudes, and motivations, can help us understand the quality and relevance of content and its potential impact @xcite . moreover, identifying individual values can be useful in conflict resolution by enabling us to comprehend the underlying beliefs and motivations of opposing viewpoints. this can facilitate finding common ground and working towards a resolution that is acceptable to all. there-fore, identifying human values has the potential to play a significant role in various fields, including psychology, sociology, marketing, and others dealing with human behavior and communication. in this paper, we propose a transformer-based language model (lm) system for the valueeval task, which utilizes second-phase pre-training in an one-versus-all (ova) setting to identify the human values expressed in arguments. our approach combines both data and algorithm adaptation concepts, whereby we second-phase pre-train an lm to better adapt to the domain and transform the data to better represent the task. to align with the nature of the task and the dataset @xcite , we implement a form of prompt engineering. this involves transforming the premise, stance, and conclusion inputs into a single sentence while replacing the stance with a predefined template. moreover, we task-adapt the roberta lm by aligning it with the masked language-modeling objective to predict the probability of each stance given an argument and conclusion. to improve the model's performance, we train multiple models for each label, based on different hyperparameters and versions of the dataset that are sampled differently. finally, we use majority voting to form the final predictions. in addition to the system description presented in this paper, we make the following observations based on our approach and experiments: firstly, we observed that second-phase pre-training in the form of task-adaptation allows the underlying lm to better represent the task in the embedding space. this leads to improved performance in identifying the human values expressed in arguments. secondly, we found that utilizing an ova approach, also known as one-versus-rest, dramatically improves performance compared to using a single multi-label classifier. finally, we observed that the effectiveness of data sampling techniques varied per label, with a subset of per-label models performing better without it. this highlights the importance of experimenting with different techniques to find the optimal approach for each label. ## background while human values have long been an important consideration in formal argumentation, this task represents the first attempt to computationally identify the values behind arguments. to that end, @xcite presented the first dataset containing the conclusion, the premise's stance towards the conclusion, and the premise itself, as show in the example in ## system overview in this section, we describe our proposed transformer-based system for the identification of human values behind argument in detail, where, given a conclusion, a stance and a premise, the input is classified into one of the 20 pre-defined values categories. our system consists of an lm adaption (tapt pre-train), data transformation, ova training and tuning phase, as shown in figure 1 . 1038 3.1 language model alignment (task adaption task adaptation refers to the process of second phase pre-training a lm with domain-specific unlabeled data that can potentially lead to performance improvements in that specific topic or domain @xcite . towards that end, we implemented a slightly different approach where instead of aligning the lm on the task dataset, we trained the underlying language model on a different task. bert-based models are trained using two types of sentences, sentence a and sentence b with the first being a sentence of a given input sequence, while the latter is a second sentence. this approach is commonly used in question answering or text classification tasks, where the input consists of a pair of sentences, where sentence a is a question or a prompt, and sentence b is the text to be classified or used to answer the question. in some cases, sentence b might be the next sentence that follows sentence a, but in other cases, it might be a sentence that is randomly chosen from the same document as sentence a. during the pre-training phase, bert-based models are trained to learn a joint representation of both sentence a and sentence b using either masked language modeling (mlm) or next sentence prediction (nsp) tasks. in the mlm task, the model is trained to predict the masked words in sentence a, while in the nsp task, the model is trained to predict whether sentence b follows sentence a in the original text or not. we followed a supervised training approach to task adaption by training the model for the classification of the stance using the premise as sentence a and the conclusion as sentence b. based on the argument presented in in this example, the model should predict that the stance is against school prayer since the premise argues for allowing it, while the conclusion argues for prohibiting it. by training on both sentence a and sentence b, bert-based models can learn to understand the relationship between different sentences in a given text and capture the contextual meaning of the input sequence. ## results this section describes the overall results compared to the best approach and our experimental results that led to our proposed system. ## conclusion in this paper, we describe our transformer-based language model system for the valueeval task, which utilizes second-phase pre-training in an one-versus-all (ova) setting to identify the human values expressed in arguments. we task-adapt the roberta lm to the domain by training the model to predict the stance that connects the conclusion to the premise. furthermore, we transform the input data to better capture the semantic and contextual information in a continuous way, by replacing the stance with a connecting phrase. our system predicts based on a majority vote from predictions by an ensemble of three different sets of per label models. we show that the task-adaption improves on the systems performance, indicating the language models can learn to generate better embeddings by aligning them to this task. a possible direction for future work would be to investigate the impact of different language models as well as data transformation techniques on the systems predictive capabilities.