Sren Kierkegaard at S em E val-2023 Task 4: Label-aware text classification using Natural Language Inference (2023)

Abstract: In this paper, we describe our approach to Task 4 in SemEval 2023. Our pipeline tries to solve the problem of multi-label text classification of human values in English-written arguments. We propose a label-aware system where we reframe the multi-label task into a binary task resembling an NLI task. We propose to include the semantic description of the human values by comparing each description to each argument and ask whether there is entailment or not.

## introduction shared tasks are competitions organized by different academic institutions that put the scope on specific research problems that are challenging for the community, typically because of low data availability. natural language processing (nlp) has many shared tasks annually @xcite @xcite . in these tasks, a problem that currently concerns the community is stated, and datasets are provided to the participants, which then have to try to solve that problem to get the highest score possible, and to contribute to establishing and comparing stateof-the-art approaches. semeval 2023 is the 17th edition of the workshop, and it features 12 different nlp shared tasks, divided in several categories that include semantic structure, discourse and argumentation, medical application and social attitudes. this paper addressed task 4 which is in the field of discourse and argumentation mining. the task is a text classification task where participants have to classify whether or not an argument belongs to certain human value categories @xcite . the categories are compiled from a social science study by @xcite . the participants are able to submit runs that detect either a subset of the seven most common categories or all of them. each argument is divided into a premise text, a conclusion text and a binary stance of the premise to the conclusion, which can be either "in favour of" or "against". @xcite provide a dataset with the challenges of identifying a subset of fundamental values behind written arguments. the task is a multi-label text classification. we propose to address the task by reframing the task from a multi-label task to a binary task which includes the textual semantic information of the labels. we reformulated the task to resemble a natural language inference (nli) task. the structure of nli tasks is given a tuple with premisehypothesis to decide if there is entailment, neutrality or contradiction between the premise and hypothesis @xcite . we propose a similar structure to our task at hand; we generate pairs of each argument with a textual description of each human-value. we then set the training label to whether or not the pairs entail each other or not. the motivation is 1) this generates more training examples for a model to learn from and 2) this makes the system label-aware, and 3) it makes the opportunity to leverage on large-scale pre-trained nli models. ## background the task the task is a multi-label text classification task on identifying values behind written arguments in english. the aim is to predict any subset of human-values a written argument is building on. there are in total 20 categories of values, which are generated from a social science study @xcite . the provided dataset consists of approximately 9000 arguments based on several sources. the dataset has a structure of premise, conclusion and stance, which together create the whole textual input. an input example from the dataset is: in the dataset, the 'conclusion' and 'stance' appear multiple times as templates, whereas the 'premise' is a human-written text to support the conclusion of a given stance. hence, it is in the 'premise', we want to look for the values underlying the argument. the annotation of the dataset happen in an open-source process @xcite , where the annotators did not use the 20 value categories directly. instead, they were asked to identify 54 more "direct/simpler values" (first-level values) which later are translated into the 20 value categories ( second-level values). if any of the 54 firstlevel values are present in the argument then the corresponding second value is marked as present in the argument as well. this is supposed to be an easier/faster task for annotators to reply yes/no to whether each simpler value is present or not in an argument. hence, we also hypothesise it might be easier for a model to use how the dataset originally where annotated by including this information in the system. the training dataset contains a class imbalance which can be seen in figure 1 . label-aware modelling in our system, we wanted to include the semantic information from the labels. hence, we look to label-aware models. in classic text classification, labels are treated as a one-hot encoded vector or indices, but the language model does not process any information present in the label name, even though there is usually useful information encoded in the label names, and sometimes even a description about the label can provide information to the model. however, a recent stream of work has indeed worked on including the semantic information from the labels in the models, eg. @xcite . ## experimental setup we use the huggingface implementation of training transformer-based models @xcite . instead of focusing on the base model of roberta, we explore different transformer models pretrained over nli datasets, like snli, to fine-tune a model already good at an nli task. however, we avoid really large models, to be mindful of computational resources and to be able to train in a manageable amount of time. we chose to experiment with the available model pepa/roberta-base-snli foot_0 which is a version of roberta base, fine-tuned over the snli dataset. test set / approach all self-direction: thought self-direction: action stimulation hedonism achievement power: dominance power: resources face security: personal security: societal tradition conformity: rules conformity: interpersonal humility benevolence: caring benevolence: dependability universalism: concern universalism: nature universalism: tolerance universalism: objectivity main best per category .59 .61 .71 .39 .39 .66 .50 .57 .39 .80 .68 .65 .61 .69 .39 .60 .43 .78 .87 .46 .58 best approach .56 .57 .71 .32 .25 .66 .47 .53 .38 .76 .64 .63 .60 .65 .32 .57 .43 .73 .82 .46 .52 bert .42 .44 .55 .05 .20 .56 .29 .44 .13 .74 .59 .43 .47 .23 .07 .46 .14 .67 .71 .32 .33 1-baseline .26 .17 .40 .09 .03 .41 .13 .12 .12 .51 .40 .19 .31 .07 .09 .35 .19 .54 .17 .22 .46 2023-01-25-15-47-28 .49 .53 .58 .19 .30 .58 .35 .50 .27 .75 .62 .59 .53 .58 .18 .54 .15 .73 .77 .38 .39 ## results we submitted our system to the official leaderboard. it achieves an overall f1 score of 0.49, being the 13th best-performing model over the main test dataset. the results per category, and also the com- the model performs better in the categories that have more instances, as there is a class imbalance in the dataset (see figure 1 ). therefore, we conclude that the critical factor in the model performance is the number of instances per category, especially due to the high number of classes and their multilabel aspect. ## conclusion we have submitted predictions to all value categories in task 4 semeval 2023. we have proposed reforming the multi-label task into a binary task and making the system label-aware. we have cast the problem to resemble an nli task and thereby be able to transfer abilities from already pre-trained backbone models for this task. we have performed hyper-parameter tuning to find the best set for our system. our submission rank above the standard bert baseline and is also in the best half of all submission in terms of overall macro f1. however, we notice some limitations in our system and set of experiments: first 1) we notice a class imbalance in the training data which we do not mitigate well. we expect different sampling strategies might help. second, 2) we do not report ablation studies. it could be relevant to examine how much e.g the reframing improves the performance and what performance boost comes from using a pre-trained nli model as a backbone.