Identifying the Human Values behind Arguments (2022)

Abstract: This paper studies the (often implicit) human values behind natural language arguments, such as to have freedom of thought or to be broadminded. Values are commonly accepted answers to why some option is desirable in the ethical sense and are thus essential both in real-world argumentation and theoretical argumentation frameworks. However, their large variety has been a major obstacle to modeling them in argument mining. To overcome this obstacle, we contribute an operationalization of human values, namely a multi-level taxonomy with 54 values that is in line with psychological research. Moreover, we provide a dataset of 5270 arguments from four geographical cultures, manually annotated for human values. First experiments with the automatic classification of human values are promising, with F _1 -scores up to 0.81 and 0.25 on average.

## introduction how come people disagree on the best course forward in controversial issues, even if they use the same information to form their opinion? a way to get to the bottom of such disagreement is to repeatedly ask them why they see something as desirable. we observe that people have different beliefs and priorities of what is generally worth striving for (e.g., personal achievements vs. humility) and how to do so (e.g., being self-directed vs. respecting traditions), often referred to as (human) values @xcite . some values tend to conflict and others to align (see figure 1 ), which can cause disagreement on the best course forward, but also the support, if not formation, of political parties that promote the respective highly revered values. moreover, one can observe different value priorities between cultures and disagreement thereon. categories that tend to conflict are placed on opposite sites. illustration adapted from @xcite . due to their outlined importance, human values are studied both in the social sciences @xcite and in formal argumentation @xcite for decades. according to the social sciences, a "value is a (1) belief (2) pertaining to desirable end states or modes of conduct, that (3) transcends specific situations, (4) guides selection or evaluation of behavior, people, and events, and (5) is ordered by importance relative to other values to form a system of value priorities." as schwartz continues, these features "make it possible to conclude that security and independence are values, whereas thirst and a preference for blue ties are not." consider the following example: "social media is good for us. though it might make people less polite, it makes our lives much easier." to understand the pragmatics of this argument, a reader has to acknowledge the belief (point 1 in the definition above) that the "end state" (2) of having a comfortable life is desirable in general (3). to concur with the statement (4), the reader further has to prefer having a comfortable life over being polite (5)-ignoring other arguments on the topic for the sake of the example. within computational linguistics, human values thus provide the context to categorize, compare, and evaluate argumentative statements, creating several possibilities: to inform social science research on values through large-scale datasets; to assess argumentation with respect to scope and strength; to generate or select arguments based on the value system of a target audience; and to identify opposing and shared values on both sides of a controversial topic. however, the task to identify values in arguments seems daunting due to their large number, often implicit use in arguments, and vague definitions. on the other hand, the creation of larger argumentation datasets, advancements in natural language understanding, and the decade-long rigorous taxonomization of values by social scientists has put such an automatic identification within reach. as a first endeavor on the automatic identification of values in written arguments, this paper makes three contributions: (1) a consolidated multilevel taxonomy of 54 human values taken from four authoritative cross-cultural social science studies (section 3); (2) a dataset of 5270 arguments from the us (most arguments), africa, china, and india, each of which manually annotated for all values by three annotators, corresponding to about 850k human judgments (section 4); and (3) first classification results per taxonomy level, establishing a baseline and revealing promising results both within and across cultures (section 5). ## background human values are of concern to most if not to all social sciences @xcite and have also been integrated into computational frameworks of argumentation @xcite . in nlp, values have been analyzed for personality profiling @xcite , but not yet for argument mining, as considered here. ## a dataset of values behind arguments this section presents the first dataset for studying human values behind arguments. each of the 5270 arguments included was annotated by three crowdworkers for all 54 values from section 3. the dataset, taxonomy description, and annotation interface are available online as webis-argvalues-22. foot_4 ## identifying values behind arguments this section presents a first attempt at automatically identifying human values using standard approaches. the first experiment focuses on the usa dataset part alone, the second on a crosscultural setting. we compare three approaches, for which we provide our implementation online: foot_9 bert. fine-tuned multi-label bert-base-uncased with batch size 8 and learning rate 2 -5 (20 epochs). svm. a linear kernel scikit-learn support vector machine trained label-wise with @xmath0. 1-baseline. classifies each argument as resorting to all values. thus always achieves a recall of 1. our evaluation focuses on the label-wise f 1score and its mean over all labels (macro-average), as well as its constituents precision and recall. we report accuracy for completeness, though the heavily skewed label distribution makes it less suited. the evaluation employs macro-averages for all metrics to give the same weight to all values. note that the 1-baseline is especially strong for the f 1 -score since it always achieves a recall of 1. by definition this baseline achieves at least as high-and in most cases higher-f 1 -scores than label-wise random guessing according to the label frequency. for calculating the p-values when comparing approaches we employ the wilcoxon signed rank significance test @xcite . as detailed in section 4, most arguments actually have both labels of the base dichotomies (levels 4a and b) assigned to them, so we do not discuss these levels deeper here. ## conclusion a computational identification of human values behind arguments is a challenging but also necessary task. with our research we contribute (1) a multilevel taxonomy with 54 values based on social science research, (2) a labeled dataset comprised of 5270 arguments from four sources, and (3) empirical analyses that cover multiple value granularity levels and compare different cultures. based on this work a logical next step are analyses that fully exploit relationships between labels. hierarchical classification approaches appear promising here (e.g., @xcite ; learning rules for multi-label classification (e.g., loza mencía and jannsen, 2016) can provide insights into value-relationships. moreover, the dataset should be extended to in-clude data from more cultures or territories, genres (e.g., blog posts), modalities (offline and spoken argumentation), and languages. probably an automated translation with manual assurance, as we did for the dataset's china part, may not be sufficient. though we optimized the annotation process, the argument acquisition requires a community effort to ensure the widest variety of data. employing annotators from different cultures is a requirement to analyze and mitigate potential sources of bias. a subsequent step of ranking the annotated values by importance can be beneficial for certain use cases, especially when using the higher taxonomy levels. values are a major contributor to argument strength @xcite , and the large-scale mining from web data could improve all of argument categorization, assessment, and generation. for example, matching values between arguments could be effective for both supporting and countering arguments. clearly expressing values behind arguments could avoid misunderstandings between humans and automated argumentation systems @xcite . similarly, an "objective" highlighting of common values behind arguments across political camps could be a step towards resolving seemingly fundamental disagreements. finally, the analysis of values in large-scale text corpora can also be of interest of social science scholars. how are values expressed online? combined with internet archive data, one could even analyse references to values over time. we thus hope that this work can serve as a first step towards a better understanding of how the public sees and saw human values in everyday (digital) life.