LRL _ NC at S em E val-2023 Task 4: The Touche23- G eorge-boole Approach for Multi-Label Classification of Human-Values behind Arguments (2023)

Abstract: The task ValueEval aims at assigning a sub- set of possible human value categories under- lying a given argument. Values behind argu- ments are often determinants to evaluate the relevance and importance of decisions in eth- ical sense, thereby making them essential for argument mining. The work presented here proposes two systems for the same. Both sys- tems use RoBERTa to encode sentences in each document. System1 makes use of features ob- tained from training models for two auxiliary tasks, whereas System2 combines RoBERTa with topic modeling to get sentence represen- tation. These features are used by a classifi- cation head to generate predictions. System1 secured the rank 22 in the official task rank- ing, achieving the macro F1-score 0.46 on the main dataset. System2 was not a part of official evaluation. Subsequent experiments achieved highest (among the proposed systems) macro F1-scores of 0.48 (System2), 0.31 (ablation on System1) and 0.33 (ablation on System1) on the main dataset, the Nahj al-Balagha dataset, and the New York Times dataset.

## introduction judgement and opinion of each person are typically governed by a set of principles which is based on a certain value system. gaining an understanding of human values behind a natural language argument can serve an important role in the field of social sciences, and policy making. the aim of the current task @xcite ) is to assign to each argument a set of underlying values, where each argument is given in the form of a conclusion, stance and premise of the argument. with each argument having multiple underlying values, the task effectively boils down to a multi-label classification task, since each argument can have multiple underlying values. english language touché23-valueeval dataset @xcite has been used for the present challenge. the labels for the challenge are given as a multilevel taxonomy of 54 human values @xcite , called level 1 labels which are further grouped into 20 human values categories called level 2 values. the aim of the task is to predict these level 2 values. the present work describes two systems: sys-tem1 and system2. both the systems use roberta @xcite to extract sentence encodings. system1 incorporates level 1 label information. this system also incorporates linguistic information, learned by predicting the stance of an argument presented by a conclusion based on a certain premise. system2 uses topic modeling to incorporate corpus-level semantic information. sys-tem1 was submitted for official evaluation though it was later observed that system2 achieves a higher macro f1-score as compared to system1. the participating teams were required to make their submissions on the platform tira @xcite . the official task evaluation platform ranks system1 on the main dataset at the position 22. the proposed systems either outperform or perform at par with the organizer's baseline. but, the systems struggle at capturing low frequency labels. the paper is organized as follows. section 2 discusses the task background and section 3 discusses related past works. section 4 provides a detailed system overview. the experimental setup and results are given in section 5 and section 6, respectively. the paper is concluded with section 7. the code for the proposed systems have been made available on github. foot_0 ## background the data made available by the organisers comprises of a main dataset and three supplementary datasets, namely, zhihu, nahj al-balagha and the new york times dataset. the main dataset consists of train/ validation/test counts as 5393/ 1896/ 1576. the task dataset is a collection of 9324 arguments on statements which include religious texts (nahj al-balagha), political discussions (group discussion ideas), free-text arguments (ibm-argq-rank-30kargs), newspaper articles (the new york times), community discussions (zhihu), and democratic discourse (conference on the future of europe). the supplementary zhihu dataset consists of 100 validation instances. nahj al-balagha and the new york times datasets have 279 and 80 test examples, respectively. a general instance of the dataset can be interpreted as, arguing in favor of/against @xmath0. an example of such an instance of the dataset, which has been assigned labels or value categories as security: societal and universalism: concern, is given in ## system overview this section discusses system1 and system2 as mentioned in section 1 . the first step in the systems is to generate an encoding for each instance using roberta @xcite . figure 1 and figure 2 give the architecture of the system1 and system2, respectively. ## experimental setup the experiments were carried on google colaboratory in python 3.8.10 with nvidia tesla p100 gpu. pytorch @xcite , and huggingface transformers @xcite were used as the key frameworks for the experiments. training is carried on the train + validation subsets of the main dataset. for validation, the validation subset of the supplementary zhihu dataset, has been used in order to make the system more robust. ## results ablation studies on system1 were conducted as follows. the results are given in ## conclusion the current work proposes two systems for the task of value identification in arguments. this is broadly a multi-label classification task. the proposed systems use roberta to get sentence encodings for each document. system2, combines roberta with topic modeling to get sentence representation. this is followed by a classification head to generate the label subsets. the other system, namely, system1, also makes use of features obtained from training models for auxiliary tasks. the auxiliary tasks, are that of prediction of level 1 values from the human-value taxonomy given an argument, and stance prediction given conclusion and premise of an argument.