Augustine of Hippo at S em E val-2023 Task 4: An Explainable Knowledge Extraction Method to Identify Human Values in Arguments with S uper ASKE (2023)

Abstract: In this paper we present and discuss the results achieved by the “Augustine of Hippo” team at SemEval-2023 Task 4 about human value detection. In particular, we provide a quantitative and qualitative reviews of the results obtained by SuperASKE, discussing respectively performance metrics and classification errors. Finally, we present our main contribution: an explainable and unsupervised approach mapping arguments to concepts, followed by a supervised classification model mapping concepts to human values.

## introduction argumentation is a communication act requiring a strong pragmatical component to be entirely understood by humans. this high-level language characterization encodes contextual information which, in turn, can be partitioned into linguistic, cognitive, physical, semantic, and social aspects. depending on actual and perceived context, individuals adjust their communication in order to convey a message as effectively as possible: this leads to the possibility of many concepts to remain implicit, especially those that can be deduced from context, which includes people's beliefs and values. semeval-2023 task 4 @xcite aims at exploring novel methodologies for extracting human values from argumentation texts written in english. focusing on this type of contextual information has the benefit of improving argumentation analysis and generation @xcite . our contribution as "augustine of hippo" team consists in superaske foot_0 , an explainable system made of aske (automated system for knowledge extraction) and random forest producing a binary output. explanations are made possible by associating human values to concepts extracted by aske, which are ordered by random forest's feature importance. our model has a median performance with respect to other runs submitted for the task. the work is organized as follows: section 2 provides some background about the task of human values detection from arguments and the original aske model; section 3 explains the behaviour of the whole framework; section 4 specifies the details on how the experiment was run; section 5 discusses the performances of the approach; section 6 sums up the most important takings of the research. ## background semeval-2023 task 4 is about human value detection. this task requires a classification model to recognise which among 20 values (or a subset of them) are present in a textual argument. specifically, the subset of human values considered is taken from @xcite which, in turn, discusses a selection of values inspired by @xcite , @xcite , @xcite @xcite . the authors in @xcite describe 4 levels contributing to the value taxonomy, which can be synthesized as follows: the first level contains 54 individual values; the second level specifies 20 values categories aggregating individual values; the third level contains 4 higher-order values; the last two levels specify base dichotomies foot_1 . our work focuses on extracting second-level value categories from arguments. in particular, each argument is represented as a triple containing a conclusion, a stance and a premise; an example is depicted in in the premise. finally, the conclusion conveys an idea according to the respective premise and stance. the target of classification is formulated as a vector @xmath0, 1] 20 indicating the presence/absence of a value in an argument. the task @xcite is organized as follows. the main dataset is taken from the work by @xcite , which has 8865 instances; this data is divided in three splits: the training set, the main validation set and the main test set. for validating the robustness of approaches, there is an additional labeled collection including 100 arguments from the recommendation and hotlist section of the chinese question-answering website zhihu. lastly, 279 arguments from the nahj al-balagha and 80 arguments from the new york times articles related to the coronavirus are made available as extra test sets. all arguments are written in english, even though source languages differ. additionally to these datasets, a value taxonomy is available in json format: this file contains all value categories and their respective values described through sample sentences (see listing 1 for an example). our solution is trained, validated and tested using all datasets and resources made available by the task organizers. ## system overview the framework we propose for detecting human values is a concatenation of aske and random forest @xcite , with the results of the analysis conducted by aske being employed as input for the rf model. every instance of this framework is tailored on a single human value, meaning that it solves a binary classification task. the main advantage of this framework is having two explainable models: concepts in aske are described by the terms that compose them with the corresponding definitions, while also being features of the rf model, which provides the importance for each of them in the trees. this allows to identify which are the most influential concepts and how much they affect the final predictions. despite aske being a completely unsupervised model, running in a zero-shot setting, its flexibility gives us the chance of proposing it in its supervised version, superaske, tuned for classification. first of all, we proceed fine-tuning the sentence embedding model employed for computing the vector representation of the acg entities (all-minilm-l6-v2 foot_2 ). being based on a siamese architecture, the model is fine-tuned by providing a pair of sentences and their corresponding semantic similarity. therefore, we retrieve all the premises from the training set, and all the descriptions of the human values provided by the task organizers: if a premise p is classified with a certain human value v, all the possible pairs of p and the descriptions of v are given to the model with a similarity score of 1, otherwise the similarity is set to 0. the fine-tuned embedding model is then employed to compute the vector representation of the initial acg components. aske is initialized with only one concept, representing a single human value v, associated with some dummy terms, having as definitions the ones provided in the value taxonomy. as document chunks, we consider only the premise of each argument, excluding stances and conclusions which appeared to not benefit to the final results. afterwards, the model is run as usual for multiple generations, providing it only the premises positively classified as v: in such a way, we ensure that the knowledge extracted by superaske is relevant to the human value analyzed. the final version of the acg is then exploited in order to compute the similarities between the concepts oc-curring in it and the whole set of premises, now including also the one negatively classified. the rf model is employed for the final binary classification. the observations are represented by the document chunks k, i.e. the premises; the features are the concepts c in the acg; the values are given by the similarities between k and c; the labels are the ground truth determining if a premise is classified or not with a given human value. ## experimental setup we test our model as follows. first, data must be split into training set, validation set and test set. this procedure has been performed by the task organizers, who have provided a unique split for all participants to use: arguments are divided in training set (61%), validation set (21%) and test set (18%); moreover, classes distribution among splits is the same. for each human value v, we retrieve only the set of premises classified as v and we run different configurations of superaske, changing the hyperparameters α, β and g, with α ∈ {-1, ..., 0.5} being the similarity threshold for the zero-shot classification phase, β ∈ {-1, ..., 0.5} being the similarity threshold for the terminology enrichment phase, and @xmath1, ..., 8} being the number of generations, i.e. the number of superaske cycles performed. each configuration learns a different version of the acg, with its peculiar concepts and assigned terms. the different acgs are then employed for computing the similarities between each premise in the entire dataset, considering also the one not classified as v, and each concept occurring in the acg. these similarities are used as input for the rf model, trained in order to predict the correct label for each premise w.r.t. the human value v. based on the performances of the rf model on the validation set, we pick the best configuration of hyperparameters for both models, superaske and rf. we then proceed repeating the same steps for each human value, training 20 different binary classifiers. evaluation is performed in two ways: using f1, precision and recall measures for each class independently and computing macro-averages over all categories. official evaluations have been done on tira platform @xcite . figure 1 depicts superaske performances and how they compare to other significant models. other pictures and tables reporting performance measures are in the appendix a.1. superaske's f1 is placed near the median for the majority of the value categories. though, this behaviour has some exceptions: "self-direction: thought" and "hedonism" f1 scores are in the first quartile, whereas "power: resources", "stimulation" and "humility" f1 scores are in the fourth quartile. considering precision and recall it can be noticed that our model is, on average, less precise than the median precision score; however, superaske has a higher recall, on average, than the median recall score. ## conclusion in this paper we used superaske to compete in the semeval-2023 task 4 with the goal of classifying arguments with respect to human values. even though our model has median performances compared to others, the results are still promising being it explainable and mostly unsupervised. on the explainability side, there could be possible improvements, such as inspecting decision trees rules to produce local explanation, i.e. how concepts extracted by aske are used by the rf classifier to compute predictions. moreover, we are planning to extend the aske component of superaske in order to make it work without an external knowledge base like wordnet. finally, the semeval-2023 task 4 collective results show that the problem of human value detection in arguments is still an open issue that cannot be easily solved by state of the art language models. this motivates us to develop future research on this task by studying nlp systems leveraging pragmatics or by implementing a perspectivist approach in ground truthing to model the subjectivity affecting this type of issues.