PAI at S em E val-2023 Task 4: A General Multi-label Classification System with Class-balanced Loss Function and Ensemble Module (2023)

Abstract: The Human Value Detection shared task\cite{kiesel:2023} aims to classify whether or not the argument draws on a set of 20 value categories, given a textual argument. This is a difficult task as the discrimination of human values behind arguments is often implicit. Moreover, the number of label categories can be up to 20 and the distribution of data is highly imbalanced. To address these issues, we employ a multi-label classification model and utilize a class-balanced loss function. Our system wins 5 first places, 2 second places, and 6 third places out of 20 categories of the Human Value Detection shared task, and our overall average score of 0.54 also places third. The code is publicly available at \url{https://www.github.com/diqiuzhuanzhuan/semeval2023}.

## introduction the human value detection shared task @xcite aims at building a system to identify human values behind a textual argument. though human values have been considered in formal argumentation for about 20 years @xcite , it is the first task to draw values from arguments computationally. the organizers provide the first dataset touché23-valueeval dataset @xcite for studying human values behind arguments. it contains 5270 arguments extracted from four different cultural backgrounds: africa, china, india, and usa. all arguments are written in english. every argument consists of three parts: a premise, a conclusion, and a stance indicating whether the premise is in favor of or against the conclusion. in this paper, we propose a general multi-label classification system for the human value detection shared task. to handle imbalanced data, we apply a class-balanced loss function and perform data augmentation for classes with fewer data. transformer-based models have demonstrated remarkable performance across a variety of nlp tasks in recent years. as a result, we adopt these models as encoder in our work. to select the best encoder model, we evaluate 4 candidates, including bert @xcite , albert @xcite , roberta @xcite , de-berta @xcite and etc. the sentence vector computed by the encoder is used as input for the multi-label classifier. we select sigmoid function as the activation function for the classifier. given the highly imbalanced nature of the data, we experiment with 5 loss functions that are specifically designed for such case, including focal loss, distribution-balanced loss and etc., and select cb-ntr loss @xcite as our final loss function. in terms of data preprocessing, we conduct rewriting and data augmentation to solve the weaknesses in the data and the model. in the end, we conduct 8-fold cross-validation and use a weighted voting ensemble based on evaluation scores of each class to get the final prediction. besides the system description, we also have some interesting findings: (1) setting individual threshold for each classifier in a multi-label classification model may not be an effective approach, (2) concatenating data that does not conform to human language can have a negative impact on model performance. ## background despite many years of research on the relationship between human values and arguments, identifying human values from argumentative texts computationally is a novel approach. in this task, the heart of value-based argumentation framework has been determined as a value taxonomy(or a set of values) that is both accepted and relevant by the organizers. the organizers extend the refined theory of @xcite ## our system in this section, we will illustrate how our system works. we use transformer-based model as our encoder model. we build four kinds of data modules: baselineargumentdataset, premiseargu-mentdataset, rewriteargumentdataset and aug-mentargumentdataset, apply the best loss function, and finally integrate the results from multiple data modules. we also provide an overview of the system in figure 1 . ## experimental setup the implementation is based on pytorch lightning foot_0 , transformers foot_1 , and allennlp foot_2 . during training, we set the batch size as 16, random seed as 42, learning rate as 1e-5, and warmup steps as 1000. we use the adamw optimizer @xcite and the polynomial decay scheduler with the power of 0.01. we set a maximum epoch of 25 and set patience of 3 for early stop. all experiments are run on one single tesla v100 gpu. the macro f1 is our metric. as for data splits, we directly use the official train and development set. but for the test set, we conduct 8-fold crossvalidation and ensemble 8 results to one. ## results and analysis in this section, we display our results and analyze the impact of each component through ablation studies. ## conclusion in this paper, we describe our system on how to use multi-label classification model to handle human value detection task @xcite . we win 5 first places, 2 second places, and 6 third places on the leaderboard. we analyze both the data and model characteristics in-depth and implement modifications on data processing and loss function design. these adjustments are verified through extensive evaluations. nevertheless, we don't leverage information contained within the labels. for future research, how to incorporate the information of individual labels into a multi-label classification model is also a promising direction for us.