Epicurus at S em E val-2023 Task 4: Improving Prediction of Human Values behind Arguments by Leveraging Their Definitions (2023)

Abstract: We describe our experiments for SemEval-2023 Task 4 on the identification of human values behind arguments (ValueEval). Because human values are subjective concepts which require precise definitions, we hypothesize that incorporating the definitions of human values (in the form of annotation instructions and validated survey items) during model training can yield better prediction performance. We explore this idea and show that our proposed models perform better than the challenge organizers’ baselines, with improvements in macro F1 scores of up to 18%.

## introduction human values are distinct beliefs that guide human behavior @xcite . examples of such values are hedonism (i.e., seeking pleasure in life), face (i.e., maintaining recognition in society), and humility (i.e., being humble). studying human values has a long history in the social sciences and in studies of formal argumentation due to their manifold applications @xcite . for example, researchers might be interested in studying how the human values individuals subscribe to affect their charitable behavior @xcite or voting behavior @xcite . in nlp, human values can be leveraged for personality recognition @xcite , or for assessing what values are implied in online discourses @xcite . to that end, task 4 (valueeval) of the semeval 2023 competition @xcite called for participants to design nlp systems that can classify a given argument as belonging to one of 20 value categories described in @xcite . human values are inherently subjective concepts, which becomes evident in, for example, ⋆ shared first authorship. the existence of many human value taxonomies, each of which contains somewhat different and differently-defined human values (e.g. @xcite . accordingly, in any scientific study of human values, clear definitions are key. therefore, we argue that it is important to incorporate the definitions of human values into model training. our approach leverages the definitions of human values (based on survey questions and annotation instructions), which we refer to as definitional statements, in a natural language inference (nli) setup. this approach offers additional theoretical benefits, such as potentially higher model validity (i.e., more accurate encodings of human values) as well as greater prediction reliability (see §3). we showed that our approach achieved better performance than the challenge baselines. we also conducted additional post-hoc analyses to explore how prediction performance would vary by the number of definitional statements per value category. we found that even with only a few definitional statements per value category, our proposed approach achieved good performance. our code is available in a public github repository ( https://github.com/ fqixiang/semeval23task4 ). ## a measurement problem in the social sciences, detecting human values is a measurement problem and typically relies on the use of (validated) survey instruments, such as the schwartz value survey @xcite . typically, respondents are asked to rate the importance of a given value (e.g., hedonism), presented in the form of survey questions, on a numerical scale. in this way, a numerical summary per human value can be assigned to each respondent. because human values are abstract concepts that are not directly observable, the respective measurements are likely to suffer from measurement error @xcite . social scientists are, therefore, particularly concerned about the content validity and reliability of such measurements. content validity refers to an instrument fully capturing what it aims to measure @xcite , whereas reliability means that measurements are stable over time and contexts (i.e., do not suffer from large random variations) @xcite . to ensure high content validity, multiple survey questions that capture different sub-aspects of a value (e.g., hedonism) are typically used. each respondent's answers to these questions about the same value are then aggregated (e.g., averaged) to obtain a single, more reliable score. likewise, incorporating definitional statements when training a model to predict human values from arguments might help to improve the content validity of the model, as well as the reliability of the predictions @xcite . for instance, the human value "achievement" has many sub-aspects (i.e., being ambitions, having success, being capable, being intellectual, and being courageous). by incorporating these finer-grained definitions of "achievement" into model training, the model can learn to encode the full scope of this value, which can in turn help to identify arguments that entail this value. furthermore, averaging a model's predictions across multiple definitional statements of the same human value category can lead to more reliable, less random results, which is consistent with the social sciences' approach. ## results ## conclusion and discussion our models achieved higher prediction performances than the challenge owners' baselines, indicating that there is merit to using definitional statements (i.e., annotation instructions and survey questions) for predicting the human values implied in textual arguments. this aligns with previous studies that incorporated dictionary definitions or survey questions for better task performance (e.g. @xcite . furthermore, we showed that by using just a small number of definitional statements per value category, we could achieve prediction performance comparable to (and in some cases, better than) when all available definitional statements are used, while significantly reducing computational overhead. as the goal of our study was not to obtain the best performance possible, but to test the idea that incorporating definitional statements into model training would improve prediction of human values, we did not try more advanced or larger language models, or techniques that could have improved prediction performance, such as paraphrasing @xcite and ensemble learning. we also fine-tuned our models on only a limited set of hyperparameters. a reason for why even the best team's model achieved a macro f 1 score of only .551 could be the low inter-rater agreement of the annotations. the average krippendorff's α was just 0.49 for level-1 value categories (i.e., sub-values of level-2 categories) @xcite . to investigate the inter-rater agreement for the level-2 values, which are the focus of this challenge, two of the authors annotated a random sample of 100 arguments from the training data and arrived at an even lower α of just 0.31. accordingly to @xcite , α values below 0.667 reflect very poor interrater agreement and high random (measurement) error. a reason for this low inter-rater agreement might be the annotation scheme requiring annotators to classify arguments as relating to 54 values -each of which with several instructions -which can overwhelm even experienced annotators. furthermore, some values and their associated explanations seem similar, such as "power: resources"which partially concerns having wealth -and "security: personal" -which partially concerns not having debts and having a comfortable life. therefore, classifying an argument as belonging to a particular value may be more subjective than intended. improvements in the human value taxonomy and/or the annotation scheme are likely needed to yield more reliable and valid measurements of human values.