Prodicus at S em E val-2023 Task 4: Enhancing Human Value Detection with Data Augmentation and Fine-Tuned Language Models (2023)

Abstract: This paper introduces a data augmentation technique for the task of detecting human values. Our approach involves generating additional examples using metadata that describes the labels in the datasets. We evaluated the effectiveness of our method by fine-tuning BERT and RoBERTa models on our augmented dataset and comparing their F1 -scores to those of the non-augmented dataset. We obtained competitive results on both the Main test set and the Nahj al-Balagha test set, ranking 14th and 7th respectively among the participants. We also demonstrate that by incorporating our augmentation technique, the classification performance of BERT and RoBERTa is improved, resulting in an increase of up to 10.1% in their F1-score.

## introduction when it comes to arguments, different people sharing the same values may have different opinions about whether one argument is persuasive or not. one cause of this is the difference between people ordering of values used to assess arguments. within computational linguistics, human values can provide context to categorize, compare, and evaluate argumentative statements, allowing for several applications: to inform social science research on values through large-scale data sets; to assess argumentation; generate or select arguments for a target audience; and to identify opposing and shared values on both sides of a controversial topic @xcite . human value detection task @xcite is concerned with automatic classification of textual arguments to determine whether an argument draws on a specific human value. provided dataset of arguments @xcite is collected from different resources within different countries and beliefs. all arguments are presented either in english or have been translated into it. to address the problem of detecting values behind arguments, we can design a multi-label clas-sifier to determine whether an argument draws on a specific value or not. to tackle the classification problem, we propose two multi-label classifiers based on two widely used language models: 1) roberta @xcite and 2) bert @xcite ; we evaluated these models performances on multiple test sets in two scenarios: 1) fine-tuned on the main training set @xcite , 2) fine-tuned on our augmented dataset. we provide all the code necessary to replicate our work in the paper's github repository foot_0 . ## background the main challenge of identifying human values behind an argument is that in most cases the argument does not explicitly refer to the desired value. the first attempt to computationally extract human values behind an argument was done by @xcite . they first proposed a 4-level taxonomy of human values. after that they provided three baseline models to classify values in each level. for sake of this paper and this task @xcite ## system overview our proposed classifiers are heavily relied on transformer based pre-trained language models. language modeling is the task of predicting the next word given a sequence of input words. currently, state-of-the-art results in language modeling are achieved by training transformer models that use an attention mechanism to extract a representation for an input sequence (encoding) and generate the next word from the extracted representation in a separate decoding module @xcite . although language modeling differs from our classification task, we can transfer the learned knowledge about language structures from these models to our task by changing the model's head, which is the output layer of the decoder part of the transformer. instead of predicting the next word using the output embeddings of the decoders, one can classify these embeddings using a fully connected layer. it has the benefit of only training the classifier head instead of the entire model. to accomplish this task, we fine-tuned two commonly used language models, bert and roberta. we provided arguments as input and generated a binary level-2 value security: societal level-1 value have a safe country definition resulting in a stronger state created input resulting in a stronger state is an example of having a safe country ## experimental setup the task organizer divided the data into three main sets (train, validation, test) and three supplementary sets (zhihu validation, nahj al-balagha test, the new york times test) @xcite . we generated eleven versions of the main training dataset, each one uses a different up-sampling factor k. we trained each model for 10 epochs and recorded its performance after each epoch. both the bert and roberta models performed reasonably well after 10 epochs. we compared the effectiveness of augmentation with dif-2035 ferent up-sampling factors across different datasets and models (figure 1 ). in terms of the main validation set, increasing the up-sampling factor usually resulted in an increase in the f 1 -score. both the bert and roberta models were trained on an nvidia geforce gtx 1080 ti for 10 epochs. for the task leaderboard, we submitted the results of roberta fine-tuned on the augmented dataset with @xmath0. we carefully tuned the batch size, learning rate and number of traning epochs of the model with respect to both main and zhihu validation sets. we repeated the training with different initial weights five times and submitted the best results out of the five attempts for the task. our roberta results placed 14th out of 41 on the leaderboard for the main test datasets and 7th among 20 submissions for nahj al-balagha test set ( ## results we evaluated the effectiveness of our approach on the original test datasets using two models, bert and roberta, and the results are presented in regarding the nahj al-balagha test set, the roberta model's f 1 -score is 30%, which is 2% higher than @xcite and 22% higher than the zero-shot baseline. a detailed summary of our results can be found in ## conclusion in this paper, we proposed two multi-label classification systems for the human value detection task, using a fine-tuned version of bert and roberta language models. we also introduced a data aug-mentation method to improve the performance of our models. our experimental results showed that our augmentation method can increase the f 1 -score of the models. we also achieved competitive results on the main test set and nahj al-balaghafor set, ranking 14th and 7th respectively among the participants. our work demonstrates the effectiveness of using pre-trained language models for the human value detection task and the potential benefits of data augmentation methods for improving model performance. future work can explore additional augmentation methods and further improve the performance of the models by incorporating external knowledge or domain-specific information. additionally, expanding the dataset to include arguments in other languages and from diverse cultures and perspectives can enhance the robustness of the models and better reflect the real-world scenarios.