Rudolf Christoph Eucken at S em E val-2023 Task 4: An Ensemble Approach for Identifying Human Values from Arguments (2023)

Abstract: The subtle human values we acquire through life experiences govern our thoughts and gets reflected in our speech. It plays an integral part in capturing the essence of our individuality and making it imperative to identify such values in computational systems that mimic human actions. Computational argumentation is a field that deals with the argumentation capabilities of humans and can benefit from identifying such values. Motivated by that, we present an ensemble approach for detecting human values from argument text. Our ensemble comprises three models: (i) An entailment-based model for determining the human values based on their descriptions, (ii) A Roberta-based classifier that predicts the set of human values from an argument. (iii) A Roberta-based classifier to predict a reduced set of human values from an argument. We experiment with different ways of combining the models and report our results. Furthermore, our best combination achieves an overall F1 score of 0.48 on the main test set.

## introduction human values @xcite play an integral role in determining how and why people react and respond to events in a particular way @xcite . we acquire them throughout our lifetime, and they largely govern our actions and manifest through our arguments. they play a significant role in defining our individuality and show a glimpse of our inner self by revealing our belief system. motivated by why people reason and respond in a certain way, we present computational models capable of automatically identifying such perceivable values from argument text. given a set of 20 human value categories, the task @xcite comprises classifying the most likely value categories from textual arguments in english. our approach includes an ensemble comprising three models: (i) an entailment-based model for determining the human values based on their descriptions, (ii) a roberta-based classifier that predicts the set of human values from an argument. (iii) a roberta-based classifier to predict a reduced set of human values from an argument. compared to standard classification-based approaches like models (ii) and (iii), we observe superior results from the entailment-based model. we experiment with different ways of combining the models foot_0 , attaining an overall f1 score of 0.48 on the main test set, which is an improvement over the baseline of 0.42. ## background we used the standard training, validation and test splits of the shared-task dataset @xcite in english, which is based on the webis-argvalues-22 dataset @xcite . the dataset comprises 9,324 arguments (5,393 train, 1,996 validation, and 1,935 test) from 6 diverse sources, covering religious texts, political discussions, free-text arguments, newspaper editorials and online democracy platforms. each argument is manually annotated for 20 human value categories (l2) spanning 54 human values (l1). given a textual argument and a human value category, the task involves classifying whether an argument draws on that category. ## system overview our implementation comprises an ensemble of three models, where a primary model performs textual entailment and two secondary models perform classical text classification. ## experimental setup using each of the three models independently, we perform predictions on the standard shared-task validation and test sets. illustrated in figure 1 , we combine the independent results using the following four schemes to generate the final set of predictions: (i) result set 1: using the entailment model, we predict on the test set. we compare each example with all the value descriptors to check for entailment and filter out predictions less than probability 0.8. we report the value categories of the final value descriptors. (ii) result set 2: to increase the system's recall, we append the test set predictions from the baseline classifier with the result set 1. (iii) result set 3: contrary to result set 2, here we try increasing the system's precision by reporting only the intersecting labels between the result set 1 and the reduced classifier predictions. (iv) result set 4: for each test set example, we first compute the intersecting labels from the baseline and reduced classifier predictions and append with the result set 1. across all runs, we use tira @xcite as the evaluation platform. we compute and compare overall and value category-wise f1, precision, and recall for the standard validation and test sets. ## results we report our test set results in we reason that since models like roberta use large data samples for training, they learn to encode the semantics better: they can discern the meaning of an argument and associate it with the value descriptions. furthermore, roberta is pre-trained using the next-sentence prediction task and can associate relationships between sentence pairs. on the contrary, since classification-based approaches require learning domain-specific relationships, they are bottlenecked by the volume of the available training data. hence, an ensemble approach such as result set 2 and 4 leverages the best of both scenarios and yields the best results. ## conclusion computational models for automatic detection of human values from arguments is a nascent yet crucial research direction for enabling purposeful argumentation systems. here we present an ensemble comprising entailment and classification-based models for detecting human values from argument text. we experiment with different ways of combining the model outputs and attain an overall f1 score of 0.48 on the main test set, which significantly improves the baseline.