I 2 C H uelva at S em E val-2023 Task 4: A Resampling and Transformers Approach to Identify Human Values behind Arguments (2023)

Abstract: This paper presents the approaches proposedfor I2C Group to address the SemEval-2023Task 4: Identification of Human Values behindArguments (ValueEval)”, whose goal is to classify 20 different categories of human valuesgiven a textual argument. The dataset of thistask consists of one argument per line, including its unique argument ID, conclusion, stanceof the premise towards the conclusion and thepremise text. To indicate whether the argumentdraws or not on that category a binary indication (1 or 0) is included. Participants can submit approaches that detect one, multiple, or allof these values in arguments. The task providesan opportunity for researchers to explore theuse of automated techniques to identify humanvalues in text and has potential applications invarious domains such as social science, politics,and marketing. To deal with the imbalancedclass distribution given, our approach undersamples the data. Additionally, the three components of the argument (conclusion, stanceand premise) are used for training. The systemoutperformed the BERT baseline according toofficial evaluation metrics, achieving a f1 scoreof 0.46.

## introduction human values refers to the beliefs, principles and standards that individuals or groups hold to be important and worthwhile. these values guide peo-ple´s attitudes and behaviors; they can vary across cultures @xcite , communities, and individuals. according to a study of global values survey @xcite , the most widely held values across the world are a sense of community, a sense of national pride, and a desire for social order. other commonly held values include equality, respect for others, and a desire for a peaceful world. in this context, the classification of human values in textual arguments is an important task in the field of natural language processing. understanding the values that underlie an argument could provide valuable insights into people´s beliefs, attitudes, and motivations. it could also be useful in various applications such as opinion analysis @xcite , argumentation mining, emotion recognition, and persuasive technology, among others. despite its importance, the automatic classification of human values in arguments remains a challenging problem. the task requires the ability to identify values in a text, understand the argument structure, and make a binary judgment about the presence of a value in the argument. it aims to advance the state of the art in human value classification in textual arguments , these textual arguments are compiled from the social science literature and described in detail in the accompanying acl paper @xcite . valueeval had the advantage of bringing together 40 teams, taking in 112 different runs (including the competition organizers). they provided a set of labelled data to solve the task. this dataset was highly imbalanced over the 20 categories, with a large number of instances belonging to the negative class. undersampling (arefeen et al., 2020) is a common technique used in machine learning to balance the class distribution in imbalanced datasets. to tackle this problem, we employed an undersampling strategy to decrease the number of instances within the negative class. on the other hand, we conducted experiments using various combinations of the information provided for the arguments. finally, our approach uses the premise, conclusion and stance of the argument as input features. these three components provide important information about the argument to help identify the values. in this work, we utilized transfer learning techniques by fine-tuning state-of-the-art pre-trained language models, using the transformers library. this approach allowed us to leverage the knowl-edge learned from large-scale datasets and apply it to our specific task of argumentative text classification. the results of our experiments show that our approach is effective in classifying arguments based on human values. the combination of undersampling and the use of all the data given leads to improved performance compared to other methods. our findings contribute to a better understanding of how human values can be identified in arguments and have implications for a range of applications, including opinion analysis and argumentation @xcite . however, as is common in such tasks, some categories have a low number of positive instances, i.e. instances where the argument draws on that category. this low number of positive instances can pose a challenge for machine learning algorithms, as they may not have enough examples to learn from. this can result in overfitting, where the algorithm memorizes the training data instead of generalizing to new data, therefore the model may not be able to accurately predict the outcome for new instances. ## background the input data for the study consists of two tabseparated value files, "arguments-training.tsv" and "labels-training.tsv". both contains 5,394 rows, on arguments each row represents a unique argument with its id, conclusion, stance and premise, figure 1 shows an example of an argument from the training dataset. . for the label each row corresponds to the unique argument id, and one column for each of the 20 value categories, indicating whether the argument aligns with the particular value category (1) or not (0). the dataset used in this paper was extracted from the descriptions of articles in arxiv, as described in the source @xcite . ## experimental setup for our experimental setup, we used a trainvalidation-test split to evaluate the performance of our models. we split the dataset into 80% for training, 10% for validation, and 10% for testing. this allowed us to train our models on a sufficiently large amount of data while still having a separate set of data to test the models' generalization ability. we chose not to use the validation dataset during the training process because its arguments were substantially different from those in the test dataset, which could have affected the generalization ability of our models. therefore, we solely relied on the training and test datasets for all the experiments. we used the pytorch library and the transformers package to implement our models. we used the hugging face transformers library to fine-tune pre-trained transformer models for the argument classification task. tira @xcite is the platform used for the shared task that we submitted our system to. for the experiment, the models were trained with 5 epochs, 32 batch size, 64 token length. early stopping was used to avoid overfitting while training. labels like "stimulation" only left us with 198 positive examples and 4116 negatives for our training dataset. this made the identification of values a more difficult task. by balancing it to an equal number of 0s and 1s, we have observed worse results compared to the original one. then, to improve our approach which is identifying the human value, the dataset has been adjusted to bring at least 1000 arguments for training, which has shown better results as we can see in in addition to that, we have more balanced labels like "self-direction: action" with 1116 positive examples and 3198 negatives. with our undersampling approach (see ## results our system achieved performance above the bert baseline (shown in figure 2 ), as measured by official evaluation metrics, we got better results overall. but the biggest gap is in more imbalanced classes like "stimulation", "humility" and "face" where we nearly doubled the performance. ## conclusion to conclude, our system for detection of human values behind arguments achieved competitive results, as it presents better results than the baseline bert model. our approach of utilizing undersampling to balance the dataset and focusing on the positive class proved to be effective, especially given test set / approach all self-direction: thought self-direction: action stimulation hedonism achievement power: dominance power: resources face security: personal security: societal tradition conformity: rules conformity: interpersonal humility benevolence: caring benevolence: dependability universalism: concern universalism: nature universalism: tolerance universalism: objectivity main best per category .59 .61 .71 .39 .39 .66 .50 .57 .39 .80 .68 .65 .61 .69 .39 .60 .43 .78 .87 .46 .58 best approach .56 .57 .71 .32 .25 .66 .47 .53 .38 .76 .64 .63 .60 .65 .32 .57 .43 .73 .82 .46 .52 bert .42 .44 .55 .05 .20 .56 .29 .44 .13 .74 .59 .43 .47 .23 .07 .46 .14 .67 .71 .32 .33 1-baseline .26 .17 .40 .09 .03 .41 .13 .12 .12 .51 .40 .19 .31 .07 .09 .35 .19 .54 .17 .22 .46 bert (1st run) .46 .43 .60 .25 .26 .53 .30 .44 .29 .71 .57 .47 .52 .27 .22 .50 .28 .68 .70 .40 .41 roberta (2nd run) .41 .32 .58 .29 .18 .39 .26 .49 .14 .71 .58 .50 .48 .00 .07 .46 .23 .68 .73 .32 .52 bert (3rd run) .45 .42 .59 .23 .25 .58 .33 .46 .28 .70 .59 .43 .50 .28 .20 .49 .27 .70 .68 .38 .47 the nature of the competition where identifying it is more crucial than not identifying it. however, there is still room for improvement. one potential direction for future work is to explore data augmentation techniques to increase the size of our training arguments and potentially improve model performance. there is also the possibility of extending our system to other languages, which would require additional preprocessing and potentially the development of language-specific models. finally, investigating the performance of other transformerbased models and their variants on this task could also lead to better results in future researches.