S em E val-2023 Task 4: V alue E val: Identification of Human Values Behind Arguments (2023)

Abstract: Argumentation is ubiquitous in natural language communication, from politics and media to everyday work and private life. Many arguments derive their persuasive power from human values, such as self-directed thought or tolerance, albeit often implicitly. These values are key to understanding the semantics of arguments, as they are generally accepted as justifications for why a particular option is ethically desirable. Can automated systems uncover the values on which an argument draws? To answer this question, 39 teams submitted runs to ValueEval’23. Using a multi-sourced dataset of over 9K arguments, the systems achieved F1-scores up to 0.87 (nature) and over 0.70 for three more of 20 universal value categories. However, many challenges remain, as evidenced by the low peak F1-score of 0.39 for stimulation, hedonism, face, and humility.

## introduction how come people disagree on the best course forward in controversial issues, even if they use the same information to form their opinion? we observe that people have different beliefs and priorities of what is generally worth striving for (e.g., personal achievements vs. humility) and how to do so (e.g., being self-directed vs. respecting traditions), often referred to as (human) values @xcite . some values tend to conflict, others tend to align (see figure 1 ). this can cause disagreement on the best course forward, but also the support, if not formation, of political parties that promote respective highly revered values. due to their outlined importance, human values have been studied both in the social sciences @xcite and in formal argumentation @xcite for decades. according to the former, a "value is a (1) belief (2) pertaining to desirable end states or modes of conduct, that * contact: johannes.kiesel@uni-weimar.de valueeval'23 focused exclusively on value categories. categories that tend to conflict are placed on opposite sites. taxonomy and illustration are largely adapted from @xcite and subsequent works. to understand this argument, a reader has to acknowledge the belief (point 1 in the definition) that the "end state" (2) of allowing for self-directed action, as in free speech, is desirable in general (3). to concur with the statement (4), the reader further has to prefer this state, in doubt, over interpersonal conformity (politeness; 5)-ignoring other arguments on the topic for the sake of the example. the identification of human values behind arguments by means of natural language processing can assist in argument categorization, evaluation, and generation. also, it may support the research of social scientists. the identified values can enable models of audience-specific argument strength @xcite , and it can support studies of morals @xcite and frames @xcite . to advance computational methods of human value identification, we organized the valueeval'23 shared task at semeval as part of the touché series. 1 simply put, the task is to classify whether a given argument resorts to a given value (section 3). as part of the shared task, we extended our previous dataset @xcite to more than 9k arguments (section 4). the task attracted submissions from 39 teams (section 5), which we benchmarked and analyzed on our dataset (section 6). overall, the teams achieved respectable results, beating our bert baseline by a large margin on all dataset parts for all 20 value categories. ## related work a dedicated line of research studied human values extensively. @xcite first defined values as certain end states or modes of conduct that humans desire. accordingly, they introduced the value system as a prioritization over these values based on cultural, social, and personal factors. the authors developed a practical survey of 36 values distinguishing between end states and behavior. for cross-cultural analysis, @xcite derived 48 value questions from universal individual and societal needs, including concepts such as obeying all the laws and being humble. cheng and fleischmann (2010) consolidated 12 taxonomies into a "meta-inventory" with 16 values, revealing significant overlap. another effort in unifying value taxonomies is the linked open data scheme val-uenet by @xcite , though the authors do not compare the taxonomies as such. based on these taxonomies are several studies in the social sciences: see @xcite for a recent overview and practical insights (directed at policy makers). an automated detection of human values, as is the goal of this shared task, could greatly assist such analyses @xcite . some computational frameworks of argumentation consider the strength of an argument subject to the audience's preferences defined via their values. example frameworks include value-based ar-1 https://touche.webis.de/semeval23/touche23-web gumentation schemes @xcite , defeasible logic programming @xcite , and the value-based argumentation framework of @xcite . automatically identifying values in natural language arguments is an important step in operationalizing these frameworks. outside of argumentation, several works in natural language processing utilize values. for example, in the context of interactive systems, @xcite aim to tune interactive chatbased agents towards morally acceptable behavior. however, since their operationalization of values is limited to valence (good or bad) and target (self or other), the model can not explain in abstract terms why something would be acceptable or not. @xcite follow a similar approach based on human edits that change text to morally acceptable ("value-aligned") behavior. a related dataset to the ours is valuenet by @xcite , foot_0 which contains 21k one-sentence descriptions of social scenarios (taken from social-chem-101 of forbes et al. ( 2020 )) annotated for the 10 value categories of an earlier version of schwartz' value taxonomy. a major difference to our dataset are the more ordinary situations in valuenet (e.g., whether to say "i miss mom"). a conceptual difference is that while valuenet's scenario descriptions could be seen as conclusion and its "utility" annotation (-1 to +1) as stance, the link between value category and description-the premise in our data-remains implicit in valuenet. the implicit premise is a key difference: our annotations specifically target the premise, as it is the locus of ethical reasoning. ## task description we define the task of identifying the human values behind arguments as follows: given a textual argument and a human value, classify whether the argument resorts to that value or not. specifically, we employ a set of 20 value categories from our previous work @xcite in a multi-label classification setup and evaluate approaches using macro f 1 , 3 but also provided the teams with per value category f 1 -scores, precision, and recall for deeper analyses. we use macro instead of micro f 1 to weigh each category the same, no matter their frequency. figure 1 shows the employed value taxonomy, which is largely based on that of @xcite and described in full detail by @xcite . for valueeval'23, we use the 20 value categories (level 2) in a multi-label classification setup: one yes-no decision for each pair of argument and value category. we selected level 2 for the task as it is the one usually used in social science analyses (e.g., by @xcite ), whereas level 1 is mostly used for data collection (i.e., when surveying people). ## data valueeval'23 employs the touché23-valueeval dataset, which consists of 9324 arguments collected from 6 diverse sources, namely religious texts, political discussions, free-text arguments, newspaper editorials, and online democracy platforms. each argument consists of three parts: two short textspremise and conclusion-, and a stance attribute that indicates whether the premise supports the conlusion ("in favor of") or its anti-thesis ("against"). each argument was annotated by 3 crowdworkers for 54 values, which were then mapped to 20 value categories (cf. the dataset consists of two parts (cf. while the main dataset reflects a classical indomain machine learning setup, the supplementary dataset simulates an out-of-domain application. this domain difference is reflected in several aspects of the datasets. as figure 2 shows, the majority of arguments in the main dataset have more than one (= at least two) value categories assigned to them (between 88% and 92%). as for the supplementary dataset, the zhihu split shows a similar pattern (with 81% of arguments having more than in te r p e r s o n a l r u le s soci eta l p e rs o n a l r e s o u rc e s do mi nan ce h u m il it y c o n fo r m it y : tr ad ition se cu ri ty : f a c e p o w er : ben evolence: u n iv e r s a li s m : se lfdi re c ti o n : s t im u la t io n h e d o n is m a ch iev ement ca ring depend ab ilit y c o n c e r n n a t u r e to le ra n c e ob jec tivit y thou gh t a c ti o n main (train) main (validation) zhihu (validation) main (test) nahj al-balagha (test) the new york times (test) 0.0 0.1 0.2 0.3 0.4 one value category), while much fewer labels are assigned to the arguments of the new york times (41%) and the nahj al-balagha split (33%). moreover, for the latter two splits a few arguments have no value assigned (i.e., resort to no ethical judgement): 2% for nahj al-balagha and 15% for the new york times. as figure 3 shows, also the distribution of value categories is very similar within the main dataset, but quite different to the distributions for the supplementary dataset, reflecting the difference in genres and topics. ## submissions the task received submissions from 39 teams, of which 37 provided information about their approaches-including the 29 teams who submitted a paper to semeval. the task used tira 9 @xcite for evaluation. the following is a cross-sectional overview of the approaches by the 37 teams, after an overview of the top-ranked 7 https://www.zhihu.com/explore 8 https://www.nytimes.com/ 9 https://www.tira.io/task/valueeval-at-semeval2023-human-value-detection approach. for details on single approaches, refer to the papers cited in top-ranked approaches team adam smith uses an ensemble of 12 transformer-based models: deberta and roberta, both trained for either loss minimization or f 1 -score maximization on three different folds each. 11 the roberta models were pretrained on the full ibm-argq-rank-30kargs dataset @xcite , which is the source for most arguments in the main set (cf. team john arthur fine-tuned a deberta model (microsoft/deberta-v2-xxlarge) on the task's data. they represented each input as a concatenation of stance, premise, and conclusion, separated by special tokens. they also found that using separate token symbols for stance ("favour" and "against") slightly boosted classification results. they trained their model to minimize binary cross-entropy loss. the outputs were passed through a sigmoid function to make a binary prediction for each of the value categories. a single decision threshold (0.2) was used for all categories. the team observed that having more data in the training set benefits the model's performance. just adding as many as 100 arguments (they used the zhihu validation set as an addition to the training set) lifted the f 1 results from 0.53 to 0.55. team pai (theodor zwinger) used a combination of transformer models. the training data, con- team approach resources best f1-score name (and code name if different) authors tr. en. nc. val. main nahj. nyt best per value category several of those below 0.59 0.48 0.47 adam smith schroter et al. (2023) ✓ ✓ 0.56 0.40 0.27 john arthur balikas (2023) ✓ 0.55 --pai (theodor zwinger) ma (2023) ✓ ✓ 0.54 --mao zedong zhang et al. (2023) ✓ ✓ 0.53 0.32 0.32 confucius anonymous ----0.53 0.31 0.31 arthur caplan song et al. (2023) ✓ ✓ 0.51 0.28 0.32 hitachi (r. m. hare) tsunokake et al. (2023) ✓ ✓ ✓ 0.51 0.34 0.34 sutnlp (david gauthier) hematian hemati et al. (2023) ✓ ✓ 0.50 0.29 -tübingen (stanley grenz) can (2023) ✓ 0.50 0.35 -georg simmel tian wei et al. ✓ 0.50 --t. m. scanlon molazadeh oskuee et al. (2023) ✓ 0.49 --csecu-dsg (fazlur rahman) aziz et al. (2023) ✓ 0.49 0.29 -søren kierkegaard talavera cepeda et al. (2023) ✓ ✓ 0.49 --prodicus moosavi m. and eetemadi (2023) ✓ ✓ 0.48 0.30 -machamp (robert s. hartman) van der goot (2023) ✓ 0.48 0.34 0.19 andronicus of rhodes papadopoulos et al. (2023) ✓ 0.48 --rudolf christoph eucken saha and srihari (2023) ✓ ✓ ✓ 0.48 --aristoxenus zaikis et al. (2023) ✓ ✓ 0.47 0.25 -noam chomsky honda and wilharm (2023) ✓ ✓ 0.47 0.26 -tom regan koichi tanigaki ✓ 0.47 --sina (seyyed hossein nasr) ghahroodi et al. (2023) ✓ 0.47 0.25 0.24 lrl_nc (george boole) tandon and chatterjee (2023) ✓ 0.46 0.27 -epicurus fang et al. (2023) ✓ ✓ ✓ 0.46 0.27 -i2c huelva (marquis de sade) el balima cordero et al. (2023) ✓ 0.46 --lauri ingman paulissen and wendt (2023) ✓ 0.44 --augustine of hippo ferrara et al. (2023) ✓ ✓ 0.44 0.23 0.19 mary daly dong qing ✓ 0.43 --niccolò machiavelli anonymous ✓ 0.42 0.23 0.25 philippa foot anonymous ----0.42 --bert (aristotle) kiesel et al. (2022) ✓ 0.42 0.28 0.24 francis bacon hasanaliyev et al. (2023) ✓ ✓ 0.42 --tenzin gyatso kandru et al. (2023) ✓ 0.41 --teamec (johann georg walch) stefanovitch et al. (2023) 0.40 --stfx-nlp (jesus of nazareth) heavey et al. (2023) ✓ ✓ ✓ 0.40 --quintilian mopidevi and chenna (2023) ✓ ✓ 0.38 0.20 0.10 joseph fletcher ewelina gajewska 0.34 --francisco de vitoria umberto altieri ✓ 0.32 --1-baseline (aristotle) kiesel et al. (2022) 0.26 0.13 0.15 martha nussbaum anonymous ✓ 0.24 --johann friedrich herbart anonymous ✓ 0.18 --friedrich nietzsche sundharram et al. (2023) ✓ 0.01 -- employed models transformer-based models were the most dominant techniques (35/37; cf. non-classification approaches four teams formulated the task as natural language inference: søren kierkegaard, rudolf christoph eucken, jesus of nazareth, and quintilian). for example, team søren kierkegaard constructed samples from the premise and the corresponding value category with entailment or contradictory labels inferred from whether the premise has the value category. then, they fine-tuned a roberta model that was already pre-trained on the natural language inference task @xcite . one team formulated the task as question-answering: team hitachi (r. m. hare) used bart and t5 as backbone models. the predictions were made by feeding the model a yes/no question for each value category given a (premise, stance, conclusion) triple. they also experimented with formulating the task as zero-shot question-answering with chain-of-thought prompting @xcite using gpt-3 (text-davinci-003). employed data out of the 37 submissions, all used the premise as an input, 25 used the conclusion, and 24 used the stance. for example, team tübingen (stanley grenz) found that utiliz-ing the stance as part of the output to predict instead of being in the input led to better effectiveness. four teams augmented the data with our definitions of the value categories (4 approaches): team prodicus created extra instances by concatenating human values and their descriptions as input and the corresponding value category as a label; team noam chomsky used the similarity between premises and the corresponding human values as extra features. team hitachi also employed value category descriptions and formulated the task as a question-answering problem, as described above. team epicurus augments their data by adding definitional statements formulated based on several human value surveys, such as the world value survey @xcite , and rokeach value survey @xcite . special training besides team adam smith, as described above, three more teams pre-trained on other corpora: team niccolo machiavelli finetuned debertav3, and team hitachi fine-tuned roberta on valuenet @xcite , while team machamp (robert s. hartman) fine-tuned on data from the other semeval'23 tasks ("intermediate training"). however, for most semeval tasks-including this one-the intermediate training did not result in higher effectiveness. two teams employed contrastive learning: mao zedong and t. m. scanlon). team sutnlp (david gauthier) employed adversarial training to improve the generalization of their model. label semantics five submissions (mao zedong, sutnlp, hitachi, prodicus, epicurus) employed label semantics by learning it from data or using the definitions of human values. for example, team mao-zedong utilized a label-guided attention mechanism to learn label-specific representations from the input. they also utilize a contrastive loss function to pull instances with similar labels together. team sutnlp learns a semantic representation of labels by encoding them jointly with their corresponding premises and then employs adversarial training to enhance the generalization of their model. they further capture label correlations by constructing a graph over labels using a graph convolutional network @xcite , where nodes (hidden label representations) are connected if there is at least one argument in the training data that contains these labels. ## results we evaluated the submitted approaches on the touchè23-valueeval dataset, both the main and supplementary parts (section 6.1). though we allowed teams to submit predictions only for a subset of the 20 value categories, only one team did so, and not on their best run. we use two baselines from our previous work @xcite : bert fine-tuned multi-label bert-base-uncased with batch size 8 and learning rate 2 -5 (20 epochs). 1-baseline classifies each argument as resorting to all categories: always achieves a recall of 1. additionally, we report on a meta-classification study for an analysis of straightforward approachcombinations (section 6.2) and describe our efforts to disseminate the results (section 6.3). ## conclusion this paper reports on the setup, participation, results, and insights gained from the touché task on identifying human values behind arguments (val-ueeval'23), hosted as task 4 at semeval-2023. we detail the construction both of the main dataset of 8865 arguments, as well as the supplementary dataset of 459 arguments used to test model generalizability. moreover, we provide a systematic overview of the 29 papers submitted by the participants, compare their approaches, and perform an ad-hoc meta classification. through the use of tira @xcite we can directly evaluate the approaches of the 8 teams who used docker submission on new datasets for human values detection, provided they are formatted like the datasets presented here. moreover 23 teams set up an open source repository for making their code available to other researchers. given the implicit nature of human values, very promising results were achieved during the task, with f 1 -scores above 0.56 on the main test set and up to 0.87 for individual categories. however, many challenges remain, as evidenced by the low peak f 1 -score of 0.39 for stimulation, hedonism, face, and humility. the results show that, like in many other nlp tasks, transformer models could be used to great effect, and in particular ensembling approaches showed to be quite effective. the results between the main and supplementary datasets were quite different as anticipated due to their dissimilarity, yet teams managed to beat the baselines on the supplementary datasets by a wide margin despite the total lack of in-domain training data. moreover, several teams reported that using our supplementary validation set as an addition to the training set boosted results, suggesting that a larger dataset of this kind will probably assist in improving the performance of future models even further. thus, there is still room for improvement on this task, and whether progress will come from novel models or creative applications of existing techniques remains to be seen. it is good to have own ideas and interests. be creative: arguments towards more creativity or imagination be curious: arguments towards more curiosity, discoveries, or general interestingness have freedom of thought: arguments toward people figuring things out on their own, towards less censorship, or towards less influence on thoughts be choosing own goals: arguments towards allowing people to choose what is best for them, to decide on their life, and to follow their dreams be independent: arguments towards allowing people to plan on their own and not ask for consent have freedom of action: arguments towards allowing people to be self-determined and able to do what they want have privacy: arguments towards allowing for private spaces, time alone, and less surveillance, or towards more control on what to disclose and to whom have an exciting life: arguments towards allowing people to experience foreign places and special activities or having perspective-changing experiences have a varied life: arguments towards allowing people to engage in many activities and change parts of their life or towards promoting local clubs (sports, ...) be daring: arguments towards more risk-taking have pleasure: arguments towards making life enjoyable or providing leisure, opportunities to have fun, and sensual gratification be ambitious: arguments towards allowing for ambitions and climbing up the social ladder have success: arguments towards allowing for success and recognizing achievements be capable: arguments towards acquiring competence in certain tasks, being more effective, and showing competence in solving tasks be intellectual: arguments towards acquiring high cognitive skills, being more reflective, and showing intelligence be courageous: arguments towards being more courageous and having people stand up for their beliefs have influence: arguments towards having more people to ask for a favor, more influence, and more ways to control events have the right to command: arguments towards allowing the right people to take command, putting experts in charge, and clearer hierarchies of command, or towards fostering leadership have wealth: arguments towards allowing people to gain wealth and material possession, show their wealth, and exercise control through wealth, or towards financial prosperity have social recognition: arguments towards allowing people to gain respect and social recognition or avoid humiliation have a good reputation: arguments towards allowing people to build up their reputation, protect their public image, and spread reputation have a sense of belonging: arguments towards allowing people to establish, join, and stay in groups, show their group membership, and show that they care for each other, or towards fostering a sense of belonging have good health: arguments towards avoiding diseases, preserving health, or having physiological and mental well-being have no debts: arguments towards avoiding indebtedness and having people return favors be neat and tidy: arguments towards being more clean, neat, or orderly have a comfortable life: arguments towards providing subsistence income, having no financial worries, and having a prosperous life, or towards resulting in a higher general happiness have a safe country: arguments towards a state that can better act on crimes, and defend or care for its citizens, or towards a stronger state in general have a stable society: arguments towards accepting or maintaining the existing social structure or towards preventing chaos and disorder at a societal level be respecting traditions: arguments towards allowing to follow one's family's customs, honoring traditional practices, maintaining traditional values and ways of thinking, or promoting the preservation of customs be holding religious faith: arguments towards allowing the customs of a religion and to devote one's life to their faith, or towards promoting piety and the spreading of one's religion be compliant: arguments towards abiding to laws or rules and promoting to meet one's obligations or recognizing people who do be self-disciplined: arguments towards fostering to exercise restraint, follow rules even when no-one is watching, and to set rules for oneself be behaving properly: arguments towards avoiding to violate informal rules or social conventions or towards fostering good manners be polite: arguments towards avoiding to upset other people, taking others into account, and being less annoying for others be honoring elders: arguments towards following one's parents or showing faith and respect towards one's elders be humble: arguments towards demoting arrogance, bragging, and thinking one deserves more than other people, or towards emphasizing the successful group over single persons and giving back to society have life accepted as is: arguments towards accepting one's fate, submitting to life's circumstances, and being satisfied with what one has be helpful: arguments towards helping the people in one's group and promoting to work for the welfare of others in one group be honest: arguments towards being more honest and recognizing people for their honesty be forgiving: arguments towards allowing people to forgive each other, giving people a second chance, and being merciful, or towards providing paths to redemption have the own family secured: arguments towards allowing people to have, protect, and care for their family be loving: arguments towards fostering close relationships and placing the well-being of others above the own, or towards allowing to show affection, compassion, and sympathy be responsible: arguments towards clear responsibilities, fostering confidence, and promoting reliability have loyalty towards friends: arguments towards being a dependable, trustworthy, and loyal friend, or towards allowing to give friends a full backing have equality: arguments towards fostering people of a lower social status, helping poorer regions of the world, providing all people with equal opportunities in life, and resulting in a world were success is less determined by birth be just: arguments towards allowing justice to be 'blind' to irrelevant aspects of a case, promoting fairness in competitions, protecting the weak and vulnerable in society, and resulting a world were people are less discriminated based on race, gender, and so on, or towards fostering a general sense for justice have a world at peace: arguments towards nations ceasing fire, avoiding conflicts, and ending wars, or promoting to see peace as fragile and precious or to care for all of humanity be protecting the environment: arguments towards avoiding pollution, fostering to care for nature, or promoting programs to restore nature have harmony with nature: arguments towards avoiding chemicals and genetically modified organisms (especially in nutrition), or towards treating animals and plants like having souls, promoting a life in harmony with nature, and resulting in more people reflecting the consequences of their actions towards the environment have a world of beauty: arguments towards allowing people to experience art and stand in awe of nature, or towards promoting the beauty of nature and the fine arts be broadminded: arguments towards allowing for discussion between groups, clearing up with prejudices, listening to people who are different from oneself, and promoting to life within a different group for some time, or towards promoting tolerance between all kinds of people and groups in general have the wisdom to accept others: arguments towards allowing people to accept disagreements and people even when one disagrees with them, to promote a mature understanding of different opinions, or to decrease partisanship or fanaticism be logical: arguments towards going for the numbers instead of gut feeling, towards a rational, focused, and consistent way of thinking, towards a rational analysis of circumstances, or towards promoting the scientific method have an objective view: arguments towards fostering to seek the truth, to take on a neutral perspective, to form an unbiased opinion, and to weigh all pros and cons, or towards providing people with the means to make informed decisions