T übingen at S em E val-2023 Task 4: What Can Stance Tell? A Computational Study on Detecting Human Values behind Arguments (2023)

Abstract: This paper describes the performance of a system which uses stance as an output instead of taking it as an input to identify 20 human values behind given arguments, based on two datasets for SemEval-2023 Task 4. The rationale was to draw a conclusion on whether predicting stance would help predict the given human values better. For this setup—predicting 21 labels—a pre-trained language model, RoBERTa-Large was used. The system had an F$_1$-score of 0.50 for predicting these human values for the main test set while this score was 0.35 on the secondary test set, and through further analysis, this paper aims to give insight into the problem of human value identification.

## introduction arguing can take place in our everyday life to express and rationalize ideas, and the way arguments are delivered can vary in their style, language and goal @xcite . in parallel with this variation, it is stated that it is a challenge to detect values expressed in the arguments as they are not always opinionated explicitly @xcite . since argumentation is a part of our everyday life-for instance, while making decisions-it is likely that our values play a role, although this may not always be overt. therefore, a large amount of literature on human value studies is an understandable effort. with this study, i hope to contribute to computational approaches to identifying human values behind arguments. @xcite describe this current human value identification task as classifying whether a given textual argument falls into a given human value or not. out of all the experiments, the versions with stance used as a separate label in addition to the existing 20 labels (resulting in 21 labels) outperformed the rest. the underlying motivation was to mark whether the model would withdraw any information from the stance of the premise towards the conclusion. after the data pre-processing, the pretrained transformer model, roberta-large @xcite , was implemented. this approach did not turn out to perform as well on the secondary test set, nahj al-balagha, as it did on the main test set when the f 1 -scores are taken into account-corresponding scores and ranks will be discussed later in the paper in addition to other approaches. this can very well ring a bell to how daunting it can be to identify human values due to their covert usage in arguments @xcite , especially considering that the arguments in the nahj al-balagha test set are from islamic religious texts with grandiloquent content that includes sermons @xcite . the aim of this paper is to present the role of stance and a pre-trained language model to identify human values behind given arguments for semeval-2023 task 4 by @xcite by drawing attention to where the system shows no immaculateness, along with two other approaches. stanley grenz was the code name i used for the system submission. for those interested, the source code is publicly available on github. foot_0 ## background an argument is comprised of a premise (or premises) and a conclusion @xcite , and for this task, we have the additional information of stance: whether the premise is in favor of the conclusion or against it. while the object of the task still remained as human value identification, i included the stance information as a separate label and used the roberta-large model to predict the stance as well as the 20 human values, which in principle resulted in this becoming a stance detection task, too. according to @xcite , an argument generally includes a core claim with a few supporting evidence for that. as stance expresses information in terms of premise and conclusion relationship, predicting the stance could mean for the model to learn how an argument can be constructed with parallel or contrasting ideas. in addition, stance classification or detection tries to find out the position of a text towards a certain topic that is generally more abstract and may not be explicated in the text @xcite , and considering that the secondary test set, nahj al-balagha, differs in the way arguments are expressed with implicit messages, as a result of which predicting human values can become extra challenging, stance prediction could be a possible help to alleviate this challenge. i chose a pre-trained language model for this task. as @xcite states, pre-trained language models (ptms) have made great achievements in the field of nlp, which shifted the way from supervised learning approaches to "pre-training and fine-tuning". since the task in question is in line with the natural language understanding (nlu) problem (i.e. comprehending text sequences) transformerencoder-only (e.g. bert and roberta) architecture was chosen over transformer-decoder-only and transformer-encoder-decoder architectures. more specifically, the system used roberta-large. for one of the two alternative approaches described later, another transformer model, bert-large (uncased), was used so as to compare performances. while roberta-large model is a model pre-trained on english language with the masked language modelling (mlm) objective @xcite , bert is a pre-trained language model on english-with mlm and next sentence prediction goals (nsp) @xcite . @xcite points out that bert was significantly undertrained, and with certain modifications, such as training the model with bigger batches on more data and on longer sequences and removing the nsp objective, an optimized model named roberta was built. ## system overview as it was briefly mentioned earlier, the main system is based on using the stance information as an additional label rather than using it as a part of the input. for this reason, the goal turned into predicting 21 labels in total, 20 of which are the given human values. for the input, the conclusion and the premise columns in the dataset were concatenated together. while concatenating these two, i used @xmath0. the input formation can be seen in figure 1 . the input then was fed into roberta (more specifically roberta-large), and i fine-tuned this pre-trained language model, which included setting the number of iterations to 5 and the the batch size for training to 16 to predict 21 labels: predicting the 20 human values and the stance. the following sections will elaborate on the system setup and performance further. ## results and discussion a glance at the published f 1 -scores for the main argument test set and the nahj al-balagha test set can give a broad idea for the performance of the main system this paper is focused on. it is noticeable that the model performed better on the main argument test set than the secondary test set: f 1 = 0.50-ranked ninth, with 0.56 being the highest score-compared to f 1 = 0.35-ranked second, with 0.40 being the highest. there are 10 labels on the main test set where f 1score is lower than 0.50, with stimulation having the lowest score of 0.10, followed by humility with a score of 0.12. the top score is seen in security: personal with 0.77. for the secondary test set, the label that got the highest f 1 -score is achievement with 0.66 while security: personal was ranked third with 0.51. similar to the main test set, stimulation and humility are among the lowest scoring labels, with 0.00 and 0.08 respectively. in fact, 4 labels that each have f 1 = 0.00 are in this secondary test set: stimulation, power:dominance, power:resources and universalism:tolerance. another 13 labels are below 0.35. across these two datasets, one can deduce that stimulation and humility were poorly predicted. there can be several reasons for that, but one that can be already seen in @xcite is that the data frequency for these two human values is relatively low for the main dataset. however, though one can consider the influence of data frequency as a possible reason-which might be a factor for power:resources having f 1score of 0.00 for the secondary test- another question arises pertaining to the way of expressing ideas based on where the arguments were taken from. as mentioned earlier, the secondary test set has a nature of including covert messages embedded in its eloquent content, and that might be a challenge for the pre-trained models. for this, training on bigger data with more variety may help to better predict the hidden human values behind covert arguments. ## conclusion this paper presented a system to predict 20 human values behind arguments by using a pre-trained language model, roberta-large, and with specific hyperparameter tuning; the main strategy employed was to use stance as a label, which meant that the system was set to predict the human values and the stance for each argument. for comparison, two other approaches were also described in the paperone for comparing the effect of another pre-trained model and the other for the impact of including stance in the input. overall, it was shown that the main system outperformed the other two. despite the reasonable rank that the system got in semeval-2023 task 4, a close look into where the system did not perform well was worth delving into. based on the secondary test set results, it seems that language use for expressing ideas may vary depending on the content. this can be a phenomenon to investigate in a future comparative study with a bigger and a more balanced dataset, which could potentially give an insight into the performance of pre-trained models for pragmatics and see whether or not human values in arguments can be predicted better despite implicity and covertness. another suggestion is linked to transfer-learning: a model trained for ideology detection can be used on this task to see if it would contribute to the system performance on identifying human values behind arguments.