Noam C homsky at S em E val-2023 Task 4: Hierarchical Similarity-aware Model for Human Value Detection (2023)

Abstract: This paper presents a hierarchical similarity-aware approach for the SemEval-2023 task 4 human value detection behind arguments using SBERT. The approach takes similarity score as an additional source of information between the input arguments and the lower level of labels in a human value hierarchical dataset. Our similarity-aware model improved the similarity-agnostic baseline model, especially showing a significant increase in or the value categories with lowest scores by the baseline model.

## introduction people often form different arguments given the same source of information. one of the factors behind the difference lies in their "different beliefs and priorities of what is generally worth striving for (e.g., personal achievements vs. humility) and how to do so." @xcite , which is referred to as human values @xcite . this paper describes our approach to human value detection through the submission to semeval-2023 task 4. @xcite . our main strategy is adding similarity scores between the argument inputs and the human values to predict which human value categories the argument draws on. the idea of the approach is to collaborate with the hierarchical relation of the input arguments towards the lower levels of more concrete human labels. we assume that such a hierarchical information will support the model to identify the attribute of the arguments within a context of the hierarchical human value dataset. the code is available at this link. 1 in section 2, the background behind our proposed approach is described, and the approach method is explained in section 3 and section 4. ## background the goal of the task is to classify the human value categories with multi-labels, given textual arguments data with human value category labels. there are four different optional levels in the labels, and the level 2 labels serve as our prediction targets. for example, if the textual arguments would be "the leaked personal information will be defrauded by fraud gangs to gain trust and carry out fraudulent activities", our model should ideally predict the level 2 human value behind the arguments, self-direction: action, security: societal, and conformity: rules. ## system overview the baseline model is a bert embedding for premises with a classification head to output the 20 categories for human values, as proposed by @xcite . ## experimental setup we limited our experiments to the main dataset and used the provided data splits as defined in @xcite . for system development and hyperparameter tuning, we trained on the train set and evaluated on the validation set. for the final submission, we retrained the model with the best hyperparameters on both train and validation set. the best hyperparameters found can be seen in appendix a. for the fine-tuning of sbert, we combined each premise with each of the 54 concatenated examples representing the level 1 labels (as described in section 3). we assigned a gold label of 1 where the premise draws on the value and a label of -1 where it does not. this is arguably skewing the meaningfulness of the similarity score as -1 generally implies opposite meaning, whereas the different values are often independent, making 0 the better label. we did however find that for our purpose labeling these cases as -1 showed the best overall model improvement. the sbert model was finetuned for 20 epochs on the train set and evaluated on the validation set. evaluation was done with label-wise and macroaverage f 1 -scores, matching both @xcite and the official semeval evaluation. for training, the transformers foot_0 library was used @xcite . every model was trained for 20 epochs, evaluated at the end of each epoch and only the best model according to the evaluation metric was kept. ## results for the main quantitative findings, the official results of the semeval2023 competition are used. they are evaluated on the hidden test set and as such comparable to other teams. since, as of writing this paper, the test labels have not yet been published, the analysis sections use the validation split for their evaluation and their findings are thus only comparable between each other. ## conclusion since research on automated human value detection is fairly new, we decided to try an approach that to this end, we extended the bert based baseline with a fine-tuned sentence transformer model that compares representations of the premises with those of example sentences for each value. our results show a significant improvement over the baseline, especially in those categories that the baseline model failed to recognize. similarity score as an input feature however does not seem to stand on its own and should be considered an auxiliary module only. overall this was a successful project to show on another domain that sentence similarity can help detecting deeper meaning that isn't apparent at the surface level.