Hitachi at S em E val-2023 Task 4: Exploring Various Task Formulations Reveals the Importance of Description Texts on Human Values (2023)

Abstract: This paper describes our participation in SemEval-2023 Task 4, ValueEval: Identification of Human Values behind Arguments. The aim of this task is to identify whether or not an input text supports each of the 20 pre-defined human values. Previous work on human value detection has shown the effectiveness of a sequence classification approach using BERT. However, little is known about what type of task formulation is suitable for the task. To this end, this paper explores various task formulations, including sequence classification, question answering, and question answering with chain-of-thought prompting and evaluates their performances on the shared task dataset. Experiments show that a zero-shot approach is not as effective as other methods, and there is no one approach that is optimal in every scenario. Our analysis also reveals that utilizing the descriptions of human values can help to improve performance.

## introduction semeval-2023 task 4 involves detecting human values behind argumentative sentences in english. given a conclusion, stance, and premise, the task is to classify whether they entail each of the 20 pre-defined values @xcite . the performance improvement in this task can contribute to more precise analysis of argumentative texts using argument and opinion mining techniques. in this paper, we explore various task formulations for detecting human values in texts, including sequence classification, question answering, and question answering with chain-of-thought prompting. previous work on human value detection @xcite thus far has only investigated the effectiveness of sequence classification using bert @xcite , and it is still unknown what type of task formulation is effective for human value detection. to this end, we test a variety of pretrained language models. for sequence classification, we employ encoder-based models such as roberta @xcite and debertav3 @xcite . for question answering, we test both encoder-decoder and decoder models, including t5 @xcite , bart @xcite , and gpt-3 (textdavinci-003) @xcite . for question answering with chain-of-thought prompting, we use gpt-3 (text-davinci-003) in a zero-shot manner. we measure the detection performance of all approaches on three shared task datasets @xcite , including the two optional test datasets: nahj al-balagha and new york times. we also verify the effectiveness of three additional training strategies for sequence classification: loss weighting, pretraining on a similar corpus: val-uenet @xcite , and adding a value description to an input. in the competition 1 , our approach using roberta large trained with sequence classification and weighted loss achieved macro f1 scores of 0.51 and 0.34 on the main and new york times test datasets and was ranked 7th and 1st, respectively. for the nahj al-balagha dataset, the submitted approach using a mixture of the highest performing models with respect to each value on the validation set demonstrated a macro f1 score of 0.34 and was ranked 4th. our contributions are three-folds. ## methodology this section explains our approaches in this work. ## experimental setup data we used the official training dataset for training and the official validation dataset for monitoring generalization performance. for evaluation, we used the main, nahj al-balagha, and new york times test datasets. details of each dataset can be found in the task paper @xcite and on the website foot_2 . ## results as for the value-wise f1 results, the most effective approach depends on the value categories and datasets, indicating that there exists no onesize-fits-all approach for human value detection. however, we see a general trend that qa slightly outperforms sc the majority of the times. the main difference between the two is whether or not an input has a value description. thus, we hypothesize that utilizing the descriptions of human values can boost the detection performance. we verify our hypothesis in the sections 5.2 and 5.3. ## discussion and conclusion this paper has explored three task formulations for human value detection: sc, qa, and qa w/ cot, and investigated the effectiveness of each approach on the shared task dataset. experimental results demonstrated that an one-size-fits-all approach does not exist for this task, but supervised learning with the dedicated task data is necessary to obtain decent detection performance. as mentioned in section 4, we hypothesized that utilizing the descriptions of human values can boost detection performance, which was supported by the results in sections 5.2 and 5.3. although there is no one task formulation that is always effective for all value categories, different value categories can favor different task formulations. for instance, qa largely outperformed sc w/ hvd in "selfdirection: thought" and "universalism: tolerance," though the only difference between the two is in their architecture. investigating this trend in detail can be a future research direction.